Understanding and managing costs is a high priority for all enterprises, and many are using machine learning and data science to help. The problem then becomes understanding and managing the costs of the machine learning projects themselves, and how they’re contributing to the business as a whole.
Demonstrating value from these kinds of projects can be especially tricky, but with a technique we call profit-sensitive scoring, organizations can gain critical insights into the business impact that the models they build are having on an enterprise’s bottom line.
What is profit-sensitive scoring?
In profit-sensitive scoring, the costs and gains associated with both correct classifications and misclassifications are taken into account, with the goal of minimizing costs and maximizing profits.
Profit-sensitive scoring treats different types of misclassifications differently, and this is where it stands out from other methods. In data science, we tend to think of the model as doing the right thing or the wrong thing for a given case, and that’s the end of it. But in a business environment, not all mistakes are created equal.
To explore why, let’s look at a common example that we often use to explain the impact of profit-sensitive scoring: churn.
Use case example: Churn
Many companies use data analytics and machine learning to predict customer churn. In this use case, you want to be able to predict who is going to cancel a recurring subscription. Using predictive models allows companies to try and retain those customers by offering them incentives—like discounts—to stay but only if they’re caught in time.
However, mischaracterization—either labeling potential churners as happy customers who have no intention of leaving or labeling non-churners as potential churners—has associated costs.
If you label a potential churner as a happy customer, you don’t offer them any sort of discount to stay, and thus run the risk of losing their business. Likewise, if you label a non-churner as a churner, and then use that to motivate offering them a discount to stay, you’re going to be offering that discount to a happy customer.
The first case is likely to be a more costly error, as you lose all of a customer’s payments, while in the second case, you’re only losing a certain, small percentage of a happy customer’s payments.
So how do you decide what to do? That’s where profit-sensitive scoring comes in.
How profit-sensitive scoring helps
While it’s quite easy to calculate the accuracy of a model (although that’s not the end-all, be-all of model evaluation), it’s only by assigning costs to the predictions, both right and wrong, that you can tell whether your models are actually adding value to the business.
It’s sometimes the case that models that are less accurate can deliver bigger business impact when you take into account financial gains and losses. By assigning meaningful values to both properly characterized as well as mischaracterized outcomes, it’s possible to generate models that can show the impact of each. And if turns out that one type of error is more costly than others, that should be taken into account as well.
Although the churn use case example above only looked at two values, the profit-sensitive scoring technique can also be applied to examples that have more than just two types of variables.
Wrapping up
Applied correctly, the profit-sensitive technique can help you understand how your machine learning projects are supporting your business and make informed decisions about how to implement their results.
If you’d like to dive deeper into the process of profit-sensitive scoring and how to make it work for your business, check out our whitepaper Talking value.
