Understanding and managing costs is a high priority for all enterprises, and many are using machine learning and data science to help. The problem then becomes understanding and managing the costs of the machine learning projects themselves, and how they’re contributing to the business as a whole.
Demonstrating value from these kinds of projects can be especially tricky, but with a technique we call profit-sensitive scoring, organizations can gain critical insights into the business impact that the models they build are having on an enterprise’s bottom line.
What is profit-sensitive scoring?
In profit-sensitive scoring, the costs and gains associated with both correct classifications and misclassifications are taken into account, with the goal of minimizing costs and maximizing profits.
Profit-sensitive scoring treats different types of misclassifications differently, and this is where it stands out from other methods. In data science, we tend to think of the model as doing the right thing or the wrong thing for a given case, and that’s the end of it. But in a business environment, not all mistakes are created equal.
To explore why, let’s look at a common example that we often use to explain the impact of profit-sensitive scoring: churn.
Use case example: Churn
Many companies use data analytics and machine learning to predict customer churn. In this use case, you want to be able to predict who is going to cancel a recurring subscription. Using predictive models allows companies to try and retain those customers by offering them incentives—like discounts—to stay but only if they’re caught in time.
However, mischaracterization—either labeling potential churners as happy customers who have no intention of leaving or labeling non-churners as potential churners—has associated costs.
If you label a potential churner as a happy customer, you don’t offer them any sort of discount to stay, and thus run the risk of losing their business. Likewise, if you label a non-churner as a churner, and then use that to motivate offering them a discount to stay, you’re going to be offering that discount to a happy customer.
The first case is likely to be a more costly error, as you lose all of a customer’s payments, while in the second case, you’re only losing a certain, small percentage of a happy customer’s payments.
So how do you decide what to do? That’s where profit-sensitive scoring comes in.
How profit-sensitive scoring helps
While it’s quite easy to calculate the accuracy of a model (although that’s not the end-all, be-all of model evaluation), it’s only by assigning costs to the predictions, both right and wrong, that you can tell whether your models are actually adding value to the business.
It’s sometimes the case that models that are less accurate can deliver bigger business impact when you take into account financial gains and losses. By assigning meaningful values to both properly characterized as well as mischaracterized outcomes, it’s possible to generate models that can show the impact of each. And if turns out that one type of error is more costly than others, that should be taken into account as well.
Although the churn use case example above only looked at two values, the profit-sensitive scoring technique can also be applied to examples that have more than just two types of variables.
Wrapping up
Applied correctly, the profit-sensitive technique can help you understand how your machine learning projects are supporting your business and make informed decisions about how to implement their results.
If you’d like to dive deeper into the process of profit-sensitive scoring and how to make it work for your business, check out our whitepaper Talking value: Optimizing enterprise AI with profit-sensitive scoring.
Prove the worth of your machine learning projects in four easy steps
Getting buy-in on machine learning projects is hard, as is ensuring you’re making the right decision based on your model’s predictions. The best way by far to solve these common problems is to understand what your model is saying in terms of cold, hard cash. But how?
This whitepaper will show.
Additional Reading
4 Common Data Science Mistakes You Don’t Want to Make
Whether you’re new to data science or extremely experienced – mistakes happen. Here we’ll look at some of the most common data science mistakes and how to avoid them.
Gartner Magic Quadrant for Data Science and Machine Learning Platforms
Get a complimentary copy of the 2021 Gartner Magic Quadrant for Data Science and Machine Learning Platforms.
Your Path to Fully Automated Data Science
In this series of four videos RapidMiner founder, Ingo Mierswa, demonstrates a complete automated data science project from end to end.