{"id":1706,"date":"2020-09-18T21:02:15","date_gmt":"2020-09-18T21:02:15","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/09\/18\/what-is-simpsons-paradox-and-how-to-automatically-detect-it\/"},"modified":"2020-09-18T21:02:15","modified_gmt":"2020-09-18T21:02:15","slug":"what-is-simpsons-paradox-and-how-to-automatically-detect-it","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/09\/18\/what-is-simpsons-paradox-and-how-to-automatically-detect-it\/","title":{"rendered":"What is Simpson\u2019s Paradox and How to Automatically Detect it"},"content":{"rendered":"
\n

By Eric Hart, Ph.D.<\/a> and Mariam Walaa, Altair<\/b>.<\/p>\n

When we want to study relationships in data, we can plot, cross-tabulate, or model that data. When we do this, we might come across cases where the relationships we see from two different views of a single dataset lead us to opposing conclusions. These are cases of Simpson\u2019s Paradox.<\/p>\n

Finding these cases can help us understand our data better and discover interesting relationships. This article gives some examples of where these cases happen, discusses how and why they happen, and suggests ways to automatically detect these situations in your own data.<\/p>\n

\u00a0<\/p>\n

What is Simpson\u2019s Paradox?<\/h3>\n

\u00a0<\/p>\n

Simpson\u2019s Paradox refers to a situation where you believe you understand the direction of a relationship between two variables, but when you consider an additional variable, that direction appears to reverse.<\/p>\n

\u00a0<\/p>\n

Why does Simpson\u2019s Paradox happen?<\/h3>\n

\u00a0<\/p>\n

Simpson\u2019s Paradox happens because disaggregation of the data (e.g., splitting it into subgroups) can cause certain subgroups to have an imbalanced representation compared to other subgroups. This might be due to the relationship between the variables, or simply due to the way that the data has been partitioned into subgroups.<\/p>\n

Example #1: Admissions<\/strong><\/p>\n

A famous example of Simpson\u2019s Paradox appears in the admissions data for graduate school at UC Berkeley in 1973 [source<\/a>]. In this example, when looking at the graduate admissions data overall,<\/em> it appeared that men were more likely to be admitted than women (gender discrimination!), but when looking at the data for each department individually,<\/em> men were less likely to be admitted than women in most of the departments.<\/p>\n

\"\"<\/p>\n

Here is an explanation of why this happens:<\/p>\n

    \n
  1. Different departments had very different acceptance rates (some were much \u201charder\u201d to get into than others)<\/li>\n
  2. More females applied to the \u201charder” departments<\/li>\n
  3. Therefore, females had a lower acceptance rate in aggregate<\/li>\n<\/ol>\n

    This leads us to ask: which view is the correct view? Do men or women have a higher acceptance rate? Is there a gender bias in admissions at this university?<\/p>\n

    In this case, it seems most reasonable to conclude that looking at the admissions rates by department makes more sense, and the disaggregated view is correct.<\/p>\n

    Example #2: Baseball<\/strong><\/p>\n

    Another example of Simpson\u2019s Paradox can be found in the batting averages of two famous baseball players, Derek Jeter and David Justice, from 1995 and 1996 [source<\/a>]. David Justice had a higher batting average in both 1995 and 1996 individually, but Derek Jeter had a higher batting average over the two years combined.<\/p>\n

    \"\"<\/p>\n

    Here is an explanation of why this happens:<\/p>\n

      \n
    1. Both players had significantly higher batting averages in 1996 than in 1995<\/li>\n
    2. Derek Jeter had significantly more at-bats in 1996; David Justice had significantly more in 1995<\/li>\n
    3. Therefore, Derek Jeter had a higher batting average in aggregate<\/li>\n<\/ol>\n

      \"\"<\/p>\n

      Figure 1: Knowledge Studio Decision Tree displaying the imbalanced number of at-bats by each player in 1995 and 1996.<\/em><\/p>\n

      Again, we can ask: which view is the correct view? Was Derek Jeter or David Justice the better hitter? In this case, it seems most reasonable to conclude that the aggregated view is the correct view, and Derek Jeter was the better hitter over the two years.<\/p>\n

      With these two examples, it\u2019s clear why Simpson\u2019s Paradox can be an issue. It\u2019s hard to draw conclusions from data when the data tells us two opposing stories at the same time. One might be tempted to think that the disaggregated view is always better since it contains more information, but it\u2019s possible that disaggregating on an additional variable provides an unnecessary or confusing perspective.<\/p>\n

      As we see in the examples above, both cases are possible: sometimes the aggregated view is correct, and sometimes the disaggregated view is correct.<\/p>\n

      \u00a0<\/p>\n

      What to do about Simpson\u2019s Paradox<\/h3>\n

      \u00a0<\/p>\n

      Without enough domain knowledge, it\u2019s hard to know which view of the relationship between two variables makes more sense \u2013 the one with or without the third variable.<\/p>\n

      But before we think about how to deal with Simpson\u2019s Paradox, we need to find a way to efficiently detect it in a dataset. As mentioned earlier, it\u2019s possible to find an instance of Simpson\u2019s Paradox (a \u201cSimpson\u2019s Pair\u201d) simply by disaggregating a contingency table or a plot of data points and studying the results. However, there are other ways we can find Simpson\u2019s Pairs using models, e.g.:<\/p>\n

        \n
      1. By building decision trees and comparing the distributions, or<\/li>\n
      2. By building regression models and comparing the signs of the coefficients<\/li>\n<\/ol>\n

        There are benefits to both, however, this can get difficult very quickly, especially when working with big datasets. It\u2019s hard to know which variables in the dataset may reverse the relationship between two other variables, and it can be hard to check all possible pairs of variables manually. Imagine we have a dataset with only 20 variables: we\u2019d need to check almost 400 pairs to be sure to find all cases of Simpson\u2019s Paradox.<\/p>\n

        There are also further challenges to consider, even if we have searched for (and found) all possible Simpson’s Pairs. These challenges relate to interpretation, for example:<\/p>\n