Simpson's paradox in psychological sciencea practical guide

Here is a table of graduate admissions data from Berkeley in the 1970s. Overall, men were admitted at a higher rate than women, eighty-four percent versus seventy-eight percent. It looks like bias, except when you break the data down by department. Women were admitted at a higher rate than men in every single department examined. Every one. The same numbers, the same people, the same university — and the conclusion flips completely depending on how you slice it. This isn't a trick. It's a structural feature of data. Rogier Kievit and colleagues argue it's happening in psychology labs right now, in papers being published today, and most researchers have no idea. The phenomenon has a name: Simpson's paradox. The core mechanic is this — a statistical association observed at the aggregate level can reverse when you look inside the subgroups that make up that aggregate. The Berkeley case makes it concrete. When men and women applied in very different proportions to departments with very different acceptance rates, pooling everyone together created a misleading impression. Faculty A admitted women at ninety-seven percent and men at ninety-one percent. Faculty B admitted women at thirty-three percent and men at twenty percent. Women were favored in both departments. But because far more women applied to the competitive Faculty B, their overall numbers looked worse. The lurking variable — which department you applied to — was doing all the work, invisibly. Kievit and colleagues illustrate the mechanics with a hiring example too. Mr. A had an overall manuscript acceptance rate of sixty percent. Ms. B had forty percent. It looks like Mr. A is doing better. But in 2010, Ms. B accepted twenty-five percent of her papers to Mr. A's zero. In 2011, Ms. B accepted one hundred percent to Mr. A's seventy-five percent. Ms. B outperformed Mr. A in both years. The pooled number is technically accurate and completely misleading at the same time. That's the paradox — not a data error, not a calculation mistake, but an aggregation that hides the structure underneath. This starts to feel like a curiosity until you see how broadly it shows up in actual psychological research. The paper reviews examples across cognitive neuroscience, behavior genetics, clinical psychology, and intelligence research — and the pattern repeats. Take the neuroscience case. Shaw and colleagues in two thousand six studied three hundred seven developing children aged seven to eighteen, looking at whether cortical thickness, the depth of the brain's outer layer, predicted general intelligence. In the combined cross-sectional sample, there was no correlation. Zero relationship. But within age strata, the picture reversed: early childhood showed a negative correlation between cortical thickness and intelligence, while late childhood showed a moderately strong positive correlation of zero point three. A single cross-sectional study drawing from that full age range could have landed anywhere between negative and positive, depending on which ages happened to be overrepresented. A researcher with clean data, rigorous methods, and no obvious mistakes could still get the sign completely wrong. Behavior genetics offers a starker example of the real-world consequences. Herrnstein and Murray's widely cited claims about heritability — that sixty percent of IQ is heritable in any given person — were used to argue that educational programs could not equalize intelligence. Kievit and colleagues flag this as a textbook level-crossing error. Heritability is a population-level statistic. It captures patterns of co-variation between individuals in a given environment. Applying it to the question of what determines a single person's intelligence is crossing levels of explanation in a way the mathematics simply doesn't support. The policy conclusion doesn't follow from the statistic. That's not a minor interpretive slip — that's the kind of error that shapes public debates about education funding. So why do smart researchers keep making it? Kievit and colleagues are clear: this isn't about sloppy science. It's structural. Psychology routinely draws inferences that cross levels of explanation — from population patterns to subgroups, from group-level findings to individuals, from between-person correlations to within-person dynamics. And the mathematics of correlation does not preserve across those transitions. A correlation at one level genuinely does not constrain what you'll find at another. The ecological fallacy — assuming that what's true of the group is true of its members — is a close cousin. Simpson's paradox is what happens when that assumption meets data with hidden subgroup structure. The clinical literature shows the same pattern. Reid and Sullivan in two thousand twelve used a latent-class approach on data about crime victimization and offending, and found multiple clusters where the association between being a prior victim and becoming an offender ran in completely different directions across clusters — sometimes positive, sometimes negative. That explains why the literature on this topic had been producing conflicting results for years. The answer wasn't that some studies were right and others were wrong. The answer was that different samples happened to capture different subgroups with genuinely different relationships. Given all this, what should a researcher actually do? Kievit and colleagues propose a three-phase framework: prevention, diagnosis, and treatment. Prevention happens before data collection — designing studies that explicitly model the possibility of subgroup differences, rather than assuming a single pooled relationship. Diagnosis and treatment apply once you have data. The diagnostic toolkit starts simple. Visualize your data. Always. The paper is blunt about this — a good plot can be worth a thousand p-values, because it can reveal conditional reversals that summary statistics completely obscure. After that, look for sign inconsistencies between levels. In one of the paper's illustrative examples, the group-level correlation between coffee intake and neuroticism was zero point eighty-eight — strong, positive, significant. The male subgroup correlation was zero point eighty-six in the same direction. The female subgroup correlation was negative zero point eighty-five. A sign flip of that magnitude is an unmistakable red flag. Heteroscedasticity — unequal variance across the range of a variable — is another marker. When subgroup regressions run in different directions from the overall regression, the residuals tend to fan out unevenly. A Breusch-Pagan test can catch this; in the paper's illustrative case, the test statistic was eighteen point four, with a p-value below zero point zero zero one, strongly rejecting homoscedasticity. For contingency tables, conditional independence tests work the same way — a significant chi-square at the aggregate level that reverses within subgroups is a diagnostic signal. The most powerful tool in the diagnostic arsenal is cluster analysis, specifically a method called Mclust, developed by Fraley and Raftery. Mclust estimates the number of underlying components needed to explain the covariance in your data, using the Bayesian Information Criterion to avoid overfitting. Kievit and colleagues built this into an R package called Simpsons, available on CRAN, specifically designed for bivariate continuous data. The package does three things: estimates whether more than one cluster exists, fits regressions within each cluster, and runs a permutation test to ask whether any cluster's regression differs significantly from the overall group estimate. If it does, the package flags it with a warning. If the sign reverses, it explicitly announces Simpson's paradox. In the coffee and neuroticism example, the tool identified three subclusters — the overall positive association held in only one of them. The paper is careful about caveats. Clustering is sensitive to sample size and distributional assumptions. Identifying clusters doesn't automatically tell you whether the third variable is a confounder, a mediator, or something else. The tool opens the door; it doesn't walk you through it. That's the deeper point Kievit and colleagues are making. They argue that explicitly modeling for Simpson's paradox doesn't just prevent errors; it generates new understanding. When Reid and Sullivan found those opposing clusters in the victimization data, they didn't just explain why the literature was inconsistent. They revealed something real about the population — that there are qualitatively different subgroups for whom the relationship between victimization and offending runs in opposite directions. That's a finding, not just a correction. Simpson's paradox, in this framing, is an exemplar of a broader class of inferential blind spots — places where the architecture of how we collect and aggregate data systematically misleads us about the world. The mismatch between levels of explanation is where those blind spots live. A correlation between populations doesn't tell you about subgroups. A correlation between individuals doesn't tell you about developmental trajectories. A heritability estimate doesn't tell you about a single person. These aren't subtle distinctions. But they're easy to miss, especially when the data are clean and the statistics are significant and the story seems coherent. The fix isn't to distrust statistics. It's to be explicit about which level you're at, to test whether the pattern holds inside the groups that compose your sample, and to reach for the tools — visual, analytical, computational — that can surface what the aggregate is hiding. The Berkeley admissions data looked like bias until someone looked closer. A lot of psychology looks solid until you do the same. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.