{"id":8044,"date":"2020-12-29T00:26:49","date_gmt":"2020-12-29T00:26:49","guid":{"rendered":"https:\/\/healinglifespan.com\/data-science\/2020\/12\/29\/how-to-easily-check-if-your-machine-learning-model-is-fair\/"},"modified":"2020-12-29T00:26:49","modified_gmt":"2020-12-29T00:26:49","slug":"how-to-easily-check-if-your-machine-learning-model-is-fair","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/12\/29\/how-to-easily-check-if-your-machine-learning-model-is-fair\/","title":{"rendered":"How to easily check if your Machine Learning model is fair?"},"content":{"rendered":"
\n

By Jakub Wi\u015bniewski<\/a>, data science student and research software engineer in MI2 DataLab<\/b>.<\/p>\n

<\/p>\n

Photo by\u00a0Eric Krull<\/a>\u00a0on\u00a0Unsplash<\/a>.<\/em><\/p>\n

We live in a world that is getting more divided each day. In some parts of the world, the differences and inequalities between races, ethnicities, and sometimes sexes are aggravating. The data we use for modeling is, in the major part, a reflection of the world it derives from. And the world can be biased, so data and therefore the model will likely reflect that.\u00a0We propose a way in which ML engineers can easily check if their model is biased.\u00a0<\/strong>Our fairness tool now works only with classification models.<\/p>\n

\u00a0<\/p>\n

Case study<\/h3>\n

\u00a0<\/p>\n

To showcase the abilities of the\u00a0dalex fairness module<\/a>, we will be using the well-known\u00a0German Credit Data dataset<\/a>\u00a0to assign risk for each credit-seeker. This simple task may require using an interpretable\u00a0decision tree classifier<\/em>.<\/p>\n

Once we have\u00a0dx.Explainer<\/em>\u00a0we need to execute the method\u00a0model_fairness()<\/em>, so it can calculate all necessary metrics among the subgroups from the\u00a0protected\u00a0<\/em>vector, which is an array or a list with sensitive attributes denoting sex, race, nationality, etc., for each observation (individual). Apart from that, we will need to point which subgroup (so which unique element of\u00a0protected<\/em>) is the most privileged, and it can be done through\u00a0privileged\u00a0<\/em>parameter, which in our case will be older males.<\/p>\n

This object has many attributes, and we will not go through each and every one of them. A more detailed overview can be found in this\u00a0tutorial<\/a>. Instead, we will focus on one method and two plots.<\/p>\n

\u00a0<\/p>\n

So is our model biased or not?<\/h3>\n

\u00a0<\/p>\n

This question is simple, but because of the nature of bias, the response will be: it depends. But this method measuring bias from different perspectives so that no bias model can go through. To check fairness, one has to use\u00a0fairness_check()<\/em>\u00a0method.<\/p>\n

\n
fobject.fairness_check(epsilon = 0.8) # default epsilon\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
The following chunk is the console output from the code above.<\/p>\n
\n
Bias detected in 1 metric: FPR\r\n\r\nConclusion: your model cannot be called fair because 1 metric score exceeded acceptable limits set by epsilon.\r\nIt does not mean that your model is unfair but it cannot be automatically approved based on these metrics.\r\n\r\nRatios of metrics, based on 'male_old'. Parameter 'epsilon' was set to 0.8 and therefore metrics should be within (0.8, 1.25)\r\n                   TPR       ACC       PPV       FPR       STP\r\nfemale_old    1.006508  1.027559  1.000000  0.765051  0.927739\r\nfemale_young  0.971800  0.937008  0.879594  0.775330  0.860140\r\nmale_young    1.030369  0.929134  0.875792  0.998532  0.986014\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
The bias was spotted in metric FPR, which is the False Positive Rate. The output above suggests that the model cannot be\u00a0automatically\u00a0<\/strong>approved (like said in the output above). So it is up to the user to decide. In my opinion, it is not a fair model. Lower FPR means that the privileged subgroup is getting False Positives more frequently than the unprivileged.<\/p>\n
More on\u00a0fairness_check()<\/em><\/strong><\/p>\n
We get the information about bias, the conclusion, and metrics ratio raw DataFrame. There are metrics TPR (True Positive Rate), ACC (Accuracy), PPV (Positive Predictive Value), FPR (False Positive Rate), STP(Statistical parity). The metrics are derived from a\u00a0confusion matrix<\/a>\u00a0for each unprivileged subgroup and then divided by metric values based on the privileged subgroup. There are 3 types of possible conclusions:<\/p>\n
\n
# not fair\r\nConclusion: your model is not fair because 2 or more metric scores exceeded acceptable limits set by epsilon.\r\n\r\n# neither fair or not\r\nConclusion: your model cannot be called fair because 1 metric score exceeded acceptable limits set by epsilon.It does not mean that your model is unfair but it cannot be automatically approved based on these metrics.\r\n\r\n# fair\r\nConclusion: your model is fair in terms of checked fairness metrics.\r\n\r\n<\/pre>\n<\/div>\n
\u00a0<\/p>\n
A DA true fair model would not exceed any metric, but when true values (target) are dependent on sensitive attributes, then things get complicated and out of scope for this blog. In short, some metrics will not be equal, but they will not necessarily exceed the user’s threshold. If you want to know more, then I strongly suggest checking out the\u00a0Fairness and machine learning<\/a>\u00a0book, especially chapter 2.<\/p>\n
But one could ask why our model is not fair, on what grounds are we deciding?<\/strong><\/p>\n
The answer to this question is tricky, but the method of judging fairness seems to be the best so far. Generally, the score for each subgroup should be close to the score of the privileged subgroup. To put it in a more mathematical perspective, the ratios between scores of privileged and unprivileged metrics should be close to 1. The closer it is to 1, the more fair the model is. But to relax this criterion a little bit, it can be written more thoughtfully:<\/p>\n
<\/p>\n
\u00a0<\/p>\n
Where the\u00a0epsilon\u00a0<\/em>is a value between 0 and 1, it should be a minimum acceptable value of the ratio. By default, it is 0.8, which adheres to the\u00a0four-fifths<\/a>\u00a0rule (80% rule) often looked at in hiring. It is hard to find a non-arbitrary boundary between a fair and discriminative difference in metrics, and checking if the ratios of the metrics are exactly 1 would be pointless because what if the ratio is 0.99? This is why we decided to choose 0.8 as our default\u00a0epsilon\u00a0<\/em>as it is the only known value to be a tangible threshold for the acceptable amount of discrimination. Of course, a user may change this value to their needs.<\/p>\n
\u00a0<\/p>\n
Bias can also be plotted<\/h3>\n
\u00a0<\/p>\n
There are two bias detection plots available (however, there are more ways to visualize bias in the package)<\/p>\n