Teaching Simpson’s Paradox Through a Medical Study

Simpson's paradox is a statistical pattern where a trend that appears in an overall (aggregated) dataset disappears or reverses when the data is broken into subgroups. It sounds like a contradiction, but it isn't—it's a signal that the groups being compared aren't structured the same way underneath. Let's walk through it using your treatment study.

Step 1: Calculate the stratified rates (comparing within each severity group)

"Stratified" means dividing data into layers, or strata—here, mild and severe cases—and comparing treatments within each layer separately.

Mild cases:

• Treatment A: 81/87 = 93.1% recovery
• Treatment B: 234/270 = 86.7% recovery
• Treatment A wins by about 6.4 percentage points.

Severe cases:

• Treatment A: 192/263 = 73.0% recovery
• Treatment B: 55/80 = 68.8% recovery
• Treatment A wins again, by about 4.2 percentage points.

So within both subgroups, Treatment A has a higher recovery rate.

Step 2: Calculate the aggregate rates (combining everyone)

"Aggregate" means adding all patients together, ignoring severity.

• Treatment A overall: 273/350 = 78.0%
• Treatment B overall: 289/350 = 82.6%

Now Treatment B looks better by about 4.6 percentage points—the exact opposite of what we saw in each subgroup. This reversal is Simpson's paradox.

Step 3: Why does this happen?

The key is that patients were not evenly distributed across severity groups between the two treatments. Look at the group sizes:

• Treatment A: only 87 mild patients (25%) versus 263 severe patients (75%)
• Treatment B: 270 mild patients (77%) versus only 80 severe patients (23%)

Treatment B was used mostly on mild patients, who recover easily under almost any treatment. Treatment A was used mostly on severe patients, who have lower recovery chances no matter what treatment they receive. When you combine everyone into one big pool, Treatment A's overall score gets dragged down because it treated many more high-risk patients, while Treatment B's overall score gets boosted because it treated mostly low-risk patients. Severity acts as a "confounding variable"—a factor that influences both which treatment a patient received and their chance of recovery, distorting the simple comparison.

In short: the aggregate comparison mixes apples and oranges. It compares two treatments whose patient populations were fundamentally different in risk level, so the mixture ratio itself—not necessarily the treatment—drives the overall number.

Step 4: Does the stratified result prove Treatment A is causally better?

Not automatically. The stratified analysis is more informative than the aggregate one, but it does not by itself establish causation—the idea that Treatment A directly produces better recovery. We were told treatment assignment was not random, which matters enormously. If doctors chose treatments based on factors we haven't measured (not just labeled severity, but things like age, comorbidities, hospital resources, or patient preference), those same factors could independently affect recovery. This is sometimes called "confounding by indication": sicker or different patients get a different treatment for reasons tied to their prognosis, not because of the treatment's true effect.

Additional questions worth investigating:

1. How was severity classified, and is it a fine enough measure? "Mild" and "severe" are broad categories. Within "severe," Treatment A patients might have still been less critically ill on average than Treatment B's severe patients, reintroducing a subtler version of the same paradox.

2. Why were treatments assigned unevenly? If Treatment A was reserved for severe cases because doctors believed it worked better for them, that belief (right or wrong) could reflect real clinical judgment or unmeasured patient differences—both of which complicate interpretation.

3. Are there other confounders, such as age, comorbid conditions, hospital site, timing of treatment, or patient adherence, that differ between treatment groups and could also explain outcome differences?

4. Is the size of the effect clinically meaningful and statistically stable, or could it be within the range of chance given the sample sizes, especially the smaller severe-Treatment-B group (n=80)?

5. Was this an observational study or something closer to a controlled trial? Without randomization, we cannot rule out selection effects.

A practical checklist for interpreting similar claims:

• Ask whether the reported figure is an aggregate or a subgroup (stratified) result, and check whether they agree or diverge.
• Look for the sizes of the subgroups being compared; large imbalances are a red flag for Simpson's paradox.
• Identify plausible confounding variables that could affect both group assignment and outcome.
• Check whether the treatment or exposure was randomly assigned; if not, be cautious about causal language.
• Consider whether subgroups are defined precisely enough, or if hidden variation still exists within them.
• Be skeptical of headlines that report only the aggregate number without subgroup detail, and seek out the original study or full breakdown before accepting a causal claim.

Simpson's paradox is not a trick of arithmetic—it's a reminder that how data is grouped can change the story it tells, and that context, not just numbers, is essential to correct interpretation.

Simpson's Paradox is a statistical phenomenon that can feel like a magic trick. It occurs when a trend appears in different groups of data but disappears or reverses when these groups are combined. As public health professionals, understanding this paradox is crucial for correctly interpreting data and avoiding potentially harmful conclusions. Let's explore it using a medical study comparing two treatments for an illness.

A study of 700 patients reports on the effectiveness of Treatment A versus Treatment B. When we look at the combined, or aggregate, data, the conclusion seems straightforward. Of the 350 patients who received Treatment A, 273 recovered, a recovery rate of 78.0% (273/350). For Treatment B, 289 of 350 patients recovered, which is an 82.6% recovery rate (289/350). Based on this aggregate comparison, Treatment B appears to be the superior choice. But is the story that simple?

This is where a more careful analysis, known as a stratified comparison, becomes essential. Stratification means dividing the data into subgroups, or strata, to see if the overall trend holds. In this study, the most obvious subgroup is patient severity. Let's stratify the data into 'mild' and 'severe' cases and recalculate the recovery rates.

For patients with mild cases:

• Treatment A: 81 of 87 patients recovered, for a 93.1% recovery rate.
• Treatment B: 234 of 270 patients recovered, for an 86.7% recovery rate.

For patients with severe cases:

• Treatment A: 192 of 263 patients recovered, for a 73.0% recovery rate.
• Treatment B: 55 of 80 patients recovered, for a 68.8% recovery rate.

Suddenly, the picture has completely reversed. Within the mild group, Treatment A is more effective. Within the severe group, Treatment A is also more effective. This is Simpson's Paradox in action: Treatment A performs better in every subgroup, yet it appears worse when the groups are combined. How is this possible?

The reversal is created by two key factors: a lurking confounding variable and unequal group sizes. The confounding variable here is patient severity. A confounder is a factor that is associated with both the treatment administered and the outcome (recovery), distorting the true relationship between them. Severity is a classic confounder because patients with severe cases are inherently less likely to recover, regardless of the treatment they receive.

The second ingredient is the imbalanced assignment of treatments. Notice the distribution:

• Treatment A was given mostly to severe patients (263 of 350, or 75%).
• Treatment B was given mostly to mild patients (270 of 350, or 77%).

The aggregate result is misleading because it's not comparing like with like. It's comparing a treatment (A) used predominantly on the sickest patients against a treatment (B) used predominantly on patients with milder illness. Treatment A's overall average is dragged down by the large number of difficult, severe cases, while Treatment B's average is inflated by the large number of easier, mild cases. The stratified analysis corrects for this by making more appropriate comparisons: mild cases on Treatment A versus mild cases on Treatment B, and severe versus severe. In this context, the stratified results give a more accurate picture of treatment efficacy.

However, does this stratified result automatically prove that Treatment A causes better recovery? Not necessarily. While the stratified analysis is a major improvement, this is an observational study, not a randomized controlled trial (RCT). In an RCT, patients would be randomly assigned to a treatment, which helps ensure that, on average, the groups are similar in all respects except for the treatment received. Without randomization, other confounders could still be at play. To confidently infer causation, we must investigate further. Here are two critical questions to ask:

1. What was the basis for treatment assignment? We need to understand the selection process. Did doctors assign Treatment A to severe patients because they believed it was a more aggressive, last-resort option? Or perhaps it was assigned to severe patients who were younger and stronger, and thus considered better candidates for a more demanding therapy. This underlying logic could introduce significant bias.
2. Are there other unmeasured differences between the patient groups? Stratifying by severity is a good start, but it might not be enough. Were there systematic differences in age, co-existing health conditions (comorbidities), or the quality of the hospital where patients were treated? Any of these factors could be another confounder influencing the outcome.

When you encounter similar claims in news reports or research summaries, use this practical checklist to think critically:

• Identify the Overall Claim: Note the main conclusion drawn from the aggregate data (e.g., "Drug X is more effective").
• Look for Subgroup Analysis: Check if the data is broken down by relevant subgroups (strata) like age, sex, disease severity, or income level.
• Compare Overall vs. Subgroup Trends: Do the trends within the subgroups contradict the overall trend? If so, you have likely found an instance of Simpson's Paradox.
• Brainstorm Potential Confounders: Think about what lurking variables could be associated with both the exposure (e.g., the treatment) and the outcome, and which might be distributed unevenly across the groups.
• Evaluate the Study Design: Determine if the study was a randomized controlled trial (stronger evidence) or an observational study (more susceptible to confounding). This context is key to judging the strength of any causal claim.

Simpson's Paradox serves as a powerful lesson in statistical humility. It demonstrates that simply looking at overall numbers can be profoundly deceptive. By learning to stratify data and question the underlying factors that shape it, we can become more discerning consumers and producers of public health evidence.