Explain Simpson’s Paradox to Public Health Analysts

Simpson’s Paradox is a counterintuitive statistical phenomenon where a trend appears in several groups of data but disappears or reverses when these groups are combined. For public health analysts, this is a critical trap: relying on aggregate data can lead to policy decisions that are not only ineffective but potentially harmful. Let us examine this through a case study of two treatments for a health condition, categorized by case severity. Consider the following data: Treatment A succeeds in 81 of 87 mild cases (93.1%) and 192 of 263 severe cases (73.0%). Treatment B succeeds in 234 of 270 mild cases (86.7%) and 55 of 80 severe cases (68.8%). When we look at the groups individually, Treatment A is superior in both categories: it outperforms Treatment B by 6.4 percentage points in mild cases and 4.2 percentage points in severe cases. However, when we aggregate the data, the picture shifts. Treatment A has a total success rate of 273 out of 350 (78.0%), while Treatment B has a total success rate of 289 out of 350 (82.6%). Suddenly, Treatment B appears to be the more effective choice. This reversal occurs because of the hidden role of case severity and unequal group sizes. In this dataset, Treatment A was disproportionately assigned to severe cases, which are inherently harder to treat and have lower success rates overall. Treatment B was disproportionately assigned to mild cases, which have higher success rates. Because the 'easier' cases are weighted more heavily in Treatment B’s total, its overall success rate is artificially inflated. This is a classic example of a confounding variable—a third factor (severity) that influences both the treatment assignment and the outcome, masking the true relationship between the two. It is vital to distinguish between the numerical phenomenon and conclusions about causation. Neither the aggregate nor the stratified comparison automatically proves that one treatment causes better outcomes. The aggregate data is misleading because it ignores the severity distribution. Conversely, the stratified data shows that Treatment A is better for both groups, but it does not account for why the patients were assigned to those treatments in the first place. Perhaps doctors were more likely to give Treatment A to the sickest patients because they believed it was more potent, or perhaps they gave Treatment B to healthier patients because it was cheaper. Without knowing the mechanism of assignment, we cannot claim that Treatment A is universally superior. To avoid falling for this paradox in your own work, use this checklist when reviewing health data: First, ask if the data can be broken down into smaller, meaningful subgroups. If you see an overall trend, always check if that trend holds true within each subgroup. Second, identify potential confounding variables. Is there a factor, like age, severity, or socioeconomic status, that might influence both the treatment and the outcome? Third, examine the sample sizes. Are the groups balanced? If one group is significantly larger or has a different composition than the other, the aggregate average will be skewed toward the characteristics of that larger group. Finally, ask yourself: what is the mechanism of assignment? Why did some patients receive one treatment over another? Understanding the context behind the numbers is just as important as the numbers themselves. By looking beyond the aggregate, you ensure that your policy recommendations are based on a true understanding of patient outcomes rather than a statistical illusion.

Simpson's paradox is the name for a situation in which a pattern that holds inside every subgroup of a dataset disappears or reverses when the subgroups are combined. It is not a mathematical error and nothing has been miscounted. It is simply what happens when you add up percentages that were calculated on groups of very different sizes and very different underlying risk. For analysts who build policy briefs from summary tables, it is one of the most important traps to recognize.

Consider two treatments for the same condition, with patients classified as mild or severe cases.

Treatment A succeeded in 81 of 87 mild cases, a success rate of 93.1 percent. Treatment B succeeded in 234 of 270 mild cases, or 86.7 percent. Among mild cases, A wins by about 6 percentage points.

Treatment A succeeded in 192 of 263 severe cases, or 73.0 percent. Treatment B succeeded in 55 of 80 severe cases, or 68.8 percent. Among severe cases, A again wins, by about 4 percentage points.

Now combine the groups. Treatment A treated 87 plus 263, which is 350 patients, and succeeded in 81 plus 192, which is 273. That is an overall success rate of 78.0 percent. Treatment B also treated 350 patients, 270 plus 80, and succeeded in 234 plus 55, which is 289. Its overall success rate is 82.6 percent. In the combined table, B looks better by more than 4 points, even though A was better in both severity groups.

How can this be? Look at who received each treatment. Of the 350 patients given Treatment A, 263, or about 75 percent, were severe cases. Of the 350 patients given Treatment B, only 80, or about 23 percent, were severe. Severe cases are harder to treat no matter which drug is used: even the better treatment achieved only 73 percent success among them, compared with over 86 percent for either treatment among mild cases. Treatment A's overall figure is therefore dragged down because it is mostly an average of hard cases, while Treatment B's overall figure is propped up because it is mostly an average of easy cases.

Two ingredients had to be present for the reversal. First, severity had to be strongly related to the outcome, which it is. Second, severity had to be unequally distributed between the two treatment groups, which it also is. Remove either ingredient and the paradox vanishes. If both treatments had seen the same mix of mild and severe patients, the overall rates would have lined up with the subgroup rates. Statisticians call a variable like severity, which is linked both to which treatment a patient got and to how the patient fared, a confounder, from the sense of confounding or mixing together two separate influences. The overall table mixes the effect of the treatment with the effect of the case mix, and reports the blend as if it were one number.

Here is the point that matters most for policy work. The reversal is a numerical phenomenon. It tells you that the aggregate comparison cannot be trusted as a measure of the treatment itself, because it is contaminated by case mix. But it does not, on its own, tell you that the stratified comparison, the one broken down by severity, is the true answer either.

The aggregate comparison fails because the treatments were compared on different kinds of patients. The stratified comparison is better in this example because it holds severity constant. Yet it only removes the distortion from the one variable we happened to measure. Perhaps patients given Treatment A were also younger, or seen at better-equipped hospitals, or diagnosed earlier. Perhaps severity itself was judged after treatment had started, so that a treatment that made patients look worse initially was more often recorded as treating severe cases. Perhaps the doctors chose Treatment A for patients they privately expected to do well. Any of these would leave the subgroup percentages exactly as they are while changing what they mean. Deciding which breakdown gives the causal answer requires knowledge about how patients came to receive each treatment and about the order in which events happened. That is a question about the real-world process, not something the table can settle by itself. A percentage is a description of what happened to a group; a causal claim is a statement about what would have happened had the same people been treated differently, and the second cannot be read off the first without assumptions.

When you meet a reversal like this one, work through the following questions before writing a recommendation.

One: How many people are in each cell of the table, not just what percentage succeeded? Small or lopsided cells are the first warning sign.

Two: Were the compared groups similar in the factors that strongly predict the outcome, such as severity, age, or comorbidity? If not, the overall figure is comparing unlike populations.

Three: How was the treatment assigned? By randomization, by clinical judgment, by availability, or by patient choice? Non-random assignment is where confounding enters.

Four: Was the stratifying variable measured before treatment began, and could the treatment have influenced it? Adjusting for something the treatment caused can create its own distortion.

Five: What plausible factors have not been measured at all, and which direction would they push the result?

Six: Which comparison answers the actual policy question? If you are deciding which treatment to give an individual of known severity, the stratified rates are relevant. If you are forecasting outcomes for a population whose case mix you cannot change, the combined figure may still be informative, provided you understand why it differs.

Seven: Does the reversal persist when a different reasonable breakdown is used, or does it depend on one particular way of slicing the data?

The habit to build is simple: whenever a summary table is used to compare two groups, ask what else differs between those groups besides the label at the top of the column. Simpson's paradox is the sharpest reminder that the answer can flip the conclusion entirely.