Concerns about bias in machine learning have been making the news recently, and the topic is far from a simple one. If you’re feeling a bit confused, don’t worry! First, you’re not alone—this topic can be complicated, which is part of the reason that there’s so much discussion about it. Second, at least some of the confusion seems to be because people are using the term “bias” in different ways that aren’t necessarily compatible.
In this first in a series of blog posts about bias in machine learning, we’re going to explore the different ways that the term “bias” is used in the realm of data science and machine learning as the first step towards helping you better understand what bias in machine learning is—and, in future posts, how you can address it.
What is Bias Anyway?
The first thing that we need to be clear about is that, when someone in the data science world says “bias”, you can’t be sure exactly what they mean unless you delve a bit deeper. That’s because the word bias has a variety of meanings, some of which are specific to data science and some of which are used in the real world. What’s more, the different meanings can intersect in ways that aren’t obvious at first.
In this article, we’re going to focus on three different uses of the word “bias” that are the most relevant to machine learning, but be aware that there are other kinds of bias in the world that might or might not interact with machine learning—for example, more specific instances of what we discuss here or less related topics like cognitive bias in psychology.
It’s also possible that there are types of bias that no one in the field is yet aware of, and that bias is operating at more levels than what we present here. Science is always changing and machine learning is a relatively new field (at least in terms of its application in the business world), so we could end up being wrong a few years down the road—we’ll have to see how the science develops.
With that discussion out of the way, let’s look at the three main things someone might mean when they refer to a biased model—real-world bias, sampling bias, and model bias.
Real-world bias
If you aren’t a data scientist and you say “bias”, you probably mean the real-world bias that we see in society, based on categories like gender, age, sexuality, race, ethnicity, etc. This bias exists independent of machine learning but can obviously interact with it, as we’ll discuss below. Because of the common understanding of this word, if you hear about bias in machine learning, it’s likely this is what someone means—that a model is perpetuating structural racism or stereotypes with its predictions.
However, in and of itself, this term isn’t terribly helpful if you’re the one who has a biased model you’re trying to fix, as it only points out that what the model’s doing is problematic, but not why it’s doing it. As we’ll see in what follows, you need to dive a bit deeper if you discover that your model is making predictions that reflect real-world bias in order to figure out why and then decide how to mitigate or understand it.
Sampling bias
Another use of the term bias in data science refers to sampling bias. If you’ve taken a statistics course, you’re familiar with this concept. It means that your data was collected in such a way that it doesn’t accurately represent the population you’re trying to study (or, in the case of machine learning, build a model to predict the behavior of).
It’s worth noting at this point that real-world bias and sampling bias are not always independent in the world of machine learning and very often interact. For example, if I run the HR department of a company that has a history of preferentially hiring men over women, and then I train a model on our past hiring decisions, the model will be biased in both of these ways—it will have had sampling bias in the training data and will exhibit real-world bias in its predictions.
Model bias
The third type of bias that you might hear about in data science is when someone says that a model is making biased predictions by overpredicting or underpredicting the probability for a given category. Note that, although this might be the result of biased data, it’s different than sampling bias in that, with sampling bias, the bias is in your data; here, the bias that’s present is in your model’s behavior.
A quick aside is warranted here to discuss the idea of algorithmic bias. We can, at a very high, abstract level, think of machine learning as involving three parts—a data set, an algorithm that creates a model, and a model that then makes predictions. The middle step, the algorithm, is very unlikely to be the source of your bias—decision trees don’t automatically favor men over women in hiring decisions, nor do convolutional neural networks prefer people who are white for college admissions.
Now, it’s possible that as more and more of these models get used, or as new types of models are rolled out, we will discover some types of bias within the algorithms that create models themselves. But for now, it is safe to say that if someone is talking about model bias, they mean the bias present in a trained model, not the algorithm that created it; that’s how we’ll use the term here.
Using the term “bias” in reference to a model’s predictions can be the trickiest to understand because some people will call a model biased when it is making skewed predictions, regardless of what that prediction is about (for example, shutting off a manufacturing process too early), even if it has no relation to real-world bias. Others, however, reserve the term “model bias” for situations where the predictions of the model match up with our definition or real-world bias—for example, preferring certain races in college admissions—regardless of whether or not the model is biased in data science terms.
Examples of Machine Learning Bias
With those definitions out of the way, let’s take a look at an example of each kind of bias we discussed, so that you can get a better idea of what these might look like in reality.
An example of sampling bias
We’ll start with sampling bias, as your data precede any model training or predictions, and because it’s often a topic that people have covered in other domains like statistics.
Let’s say that I work at a nationwide retailer in the US, and I want to build a model that offers coupons to people at check-out based on products that are frequently bought together. You pull data from your data warehouse, and start looking through it, only to discover that about 80% of the data that you have is from purchases in either Boston or New York. It’s probably intuitive that people’s buying habits in different places, especially in the city versus in more rural areas, are going to be different. If I roll out a model based on this data to the entire US, I’m going to have a problem.
Unlike the other types of bias we discuss here, you can often find data problems before you even get started with model training, as in this example. This is why data cleaning and data prep tasks, although sometimes seen as busywork for data scientists, are actually critical to the model-creation process. Sometimes, simple tools like histograms are all that you need to see that your data is really skewed in one direction and is likely to cause problems during model training. Which is one possible source for the bias in our next example—model bias.
An example of model bias
To look at pure model bias, let’s take an example that doesn’t interact with real-world bias. Let’s say that you’re a chemical manufacturer and you have a model that’s designed to predict when a certain vat is going to boil over so that you can cut off the heat source and save the product. Although it did well during training and testing, now that it’s rolled out into production, you notice that it is 400% more likely to predict boil-over events on Tuesdays.
This seems suspicious, as you can imagine—why is the model behaving this way? Well, it could be due to sampling bias. Perhaps, just by coincidence, 95% of the boil-over events that you trained your model on happened to occur on Tuesdays, and your model learned that that was a very effective way to make accurate predictions—at least in terms of the data it had available to it.
But there could be other reasons. Perhaps someone who works on Tuesdays is processing and entering data differently than others due to a misunderstanding. Or maybe on Tuesdays you get shipments of new raw materials, and having the delivery doors open causes a change in the temperature readings on the vats—the vat temperature hasn’t actually changed, but the difference is enough that the model thinks something is wrong and signals a boil-over event.
It can take some work to identify and mitigate this kind of bias if you encounter it, and doing so can often require thinking outside of data science methods.
An example of real-world bias
Examples of real-world bias are easy to find and more intuitive in many ways than the other examples here. This is because the real-word use of the term bias matches our everyday understanding, not to mention that examples of this kind of bias occasionally make the news when models with real-world bias get put into production in areas like human resources and college admissions.
For example, let’s imagine that a human resources department trains a model to score candidates who apply through their online system. After several months in production, it’s discovered that the system is giving lower scores to African American candidates, even if they have equivalent experience when compared to white applicants. If these predictions are being acted on, you’ve got real-world bias.
As noted above, this is simply a descriptive label that only identifies the problem—that the model is creating real-world bias—but not why or how you can solve it. If you’re seeing predictions like this from a model, you need to look at different potential contributing factors. Could it be sampling bias? Could it be the data that the model is getting in production? Could there be other confounding factors at play?
Addressing Biased Machine Learning Models
At this point, you’re probably wondering what to do about a biased model if you’ve found one. As mentioned, we’ll explore this topic in-depth in future posts, including how you can use RapidMiner to explore what your model is doing, but if you discover that a model is biased, there are a few things that you can do now to try and figure out why.
The first is something we’ve already talked about—look at your data. Sampling bias is one of the most common reasons that a model ends up biased, and if you go back to your data with information about the kinds of predictions that your model is biased on, you might see patterns in that original data you didn’t realize were present. In addition to looking at histograms, you can also calculate feature weights to see what columns are having the most impact on your model’s predictions.
Examining your data can be seen as a small part of general efforts towards model explainability, which help you understand what your model is doing and why—and are thus critical in figuring out why your model is making biased predictions.
Depending on what kinds of decisions the model is making, you might want to consider outside factors as well. These are things that might be causing your model to make strange predictions—like the factors we discussed above regarding boil-over events on Tuesdays—including where the data is coming from and how it’s being processed before the model sees it to render a prediction.
Bias Interactions
As has been noted already, it is often the case that these different kinds of bias interact, which is part of the reason for the confusion that exists in how people are looking at and trying to understand model bias.
This is an important point because we don’t want you to get lost in definitions and debates about what kind of model bias you’ll be looking at if you have a problem. Identifying that there’s a problem is the first step, but if you have a model that’s making biased predictions, you don’t need to spend a lot of time trying to figure out what exactly to call your issue—you need to dive in and figure out where it came from so you can address it.
Wrapping Up and Looking Forward
At this point, you should have a better understanding of the most relevant types of machine learning bias that exist, how they interact, and some ideas of how you can start looking for issues if you have a model that’s creating biased predictions.
Ultimately, any effort to mitigate bias is going to fall on humans who are looking at the data and what the models are doing to come up with a solution. That’s because bias is ultimately a societal problem, and models are only reflecting the data that they see, whether during training or in production. It thus falls to data-literate, bias-aware humans to look at what a model is doing and figure out how to improve things going forward.
Which is to say, bias isn’t just a technical problem of sampling and modeling; it’s a real-world problem that a purely technical solution won’t be enough to tackle. Only by creating awareness and building explainable models will you be able to understand what’s happening so that you can make a plan to address any bias you find.
Are you responding properly to opportunities & pressures presented by Industry 4.0?
Follow this blueprint to learn how to optimize every segment of your manufacturing business with AI.
