Skip to content

B1: The Effect-Size Reflex

This is one of the thirty thinking tools identified and explored on Hardly Thinking. Explore all the tools →

A real effect, but how big, for whom, compared to what?

The study you were waiting for

Imagine a headline about something you have distrusted for years: a food additive, a social media app, a school curriculum, a style of policing. A large study by reputable researchers has found a "significant link" between it and a harm you care about. You feel vindicated, and you send it to the friend who always argued with you about it.

Now suppose the headline had been about something you like, and the "significant link" pointed at a harm from it. You would probably ask at once how big the link is, and whether it changes anyone's odds in a way they would notice. Those questions were just as good when the study agreed with you. Most of us only think to ask them when it doesn't.

The move

The Effect-Size Reflex is a tool, which Hardly Thinking uses to mean a mental operation you can run on a claim in real time, before you believe it or pass it on. When you hear that an effect is real, ask how big it is, for whom, and compared to what.

The word "significant" causes much of the trouble. In ordinary speech it means important. In research it usually means something narrower: that a result would be unlikely if there were no real effect at all. A result can be significant in that second sense and still be tiny. A study can show, quite reliably, that something shifts an outcome by an amount no person would ever feel and no sensible decision would turn on. The size of the shift is called the effect size, and it is the number a headline about a "significant link" most often leaves out.

Running the tool takes four steps.

  1. Notice the claim that an effect is real: a "link," a "raised risk," a "significant" difference, a "proven" benefit.
  2. Ask how big it is, in units a person can feel. How many more people out of a hundred or a thousand? How many minutes, points, dollars or days?
  3. Ask for whom. Is the effect spread thinly across everyone, or concentrated in a small group where it may be large?
  4. Ask compared to what. Set the effect beside the other risks or benefits that bear on the same decision, and see whether it would change the choice anyone actually has to make.

The last step matters most. The tool does not dispute that an effect is real. It asks whether the effect is large enough to carry the conclusion being hung on it.

Why it works

A statistically real effect can still be too small to matter for any actual decision. Whether an effect exists and whether it matters are two separate questions, and public argument tends to answer the first and then behave as if it had answered the second.

This tool belongs to Inference Discipline, the family of tools that deals with errors in the step from evidence to conclusion. The evidence may be perfectly sound; the error comes when "this effect exists" becomes "this thing is dangerous" or "this thing works" without anyone checking the size of what was found.

The mistake runs in both directions: a small real effect gets inflated into a crisis by people who already feared the thing, and a large but imprecisely measured one gets dismissed as "not proven" by people who never wanted to act on it.

Statisticians have written about the gap between statistical significance and practical importance for decades. In March 2016 the American Statistical Association, a professional society of statisticians, issued a statement on p-values, the figures that usually lie behind a claim of significance. "Statistical significance is not equivalent to scientific, human, or economic significance," the statement said. It added that "any effect, no matter how tiny, can produce a small p-value if the sample size or measurement precision is high enough, and large effects may produce unimpressive p-values if the sample size is small or measurements are imprecise."

News reports can leave the size out. In a 2000 study in the New England Journal of Medicine, Ray Moynihan, then a Harkness Fellow in health-care policy, and seven colleagues examined a sample of 207 American newspaper and television stories, from 1994 to 1998, about three drugs taken to prevent disease: aspirin, the cholesterol drug pravastatin and the osteoporosis drug alendronate. Of the 207, 83 gave no figure at all for the drugs' benefits. Of the 124 that did, 103 gave only the relative benefit, such as a percentage cut in risk, and not the absolute one, which says how many people out of a hundred or a thousand would be helped.

Using it well

The legitimate use is asking how large an effect is in units a person can feel, not just whether a study found significance.

"The widespread use of digital technologies by young people has spurred speculation that their regular use negatively impacts psychological well-being," Amy Orben and Andrew Przybylski, two researchers at the University of Oxford, wrote in Nature Human Behaviour in January 2019. They then asked how big the link was. Across three large surveys of young people in Britain and the United States, they reported that the association between digital technology use and adolescent well-being "is negative but small, explaining at most 0.4% of the variation in well-being." That figure is a statistician's measure, not a unit anyone can feel. The comparison came in the university's press release, where Przybylski set the result beside other things measured in the same surveys. "Within the same dataset, we were able to demonstrate that including potatoes in your diet showed a similar association with adolescent wellbeing," he said. "Wearing corrective lenses had an even worse association."

In April 2020 the psychologists Jean Twenge, Jonathan Haidt, Thomas Joiner and W. Keith Campbell replied in the same journal that the analysis had made choices "that resulted in lower effect sizes," and they pointed to "practically important differences identified in other analyses of the same datasets, especially for social media use among girls." Their first figure used a unit a reader can picture: "Percentage of the MCS cohort with clinically relevant depressive symptoms as a function of social media use," the MCS being Britain's Millennium Cohort Study. Hardly Thinking has read only their opening paragraph and figure captions, so their numbers are not given here.

In their answer, Orben and Przybylski re-ran their analysis on the Millennium Cohort Study. Female respondents "showed a more negative association than males," they wrote, and technology use "predicted nearly one half of one percent of the variability in the well-being of girls." They reported that Twenge and colleagues "assert that percentage of variance explained is not a useful metric for evaluating the size of an effect," and answered that this measure "is a useful effect size measure in the context of our study." The teams disagreed over which unit of size answers the question.

A good use of the tool ends with the claim restated in plain units, whichever way the answer cuts.

When it becomes a weapon

The weaponized form is demanding impossible precision indefinitely as a delay tactic against acting on a real, if imprecisely sized, harm.

In one documented case, the strategy was written down. Frank Luntz, a Republican pollster and consultant, wrote a memo advising Republicans on how to talk about the environment. In March 2003, Oliver Burkeman of the Guardian reported that the memo had been produced in 2002 and obtained by the Environmental Working Group, an environmental advocacy group. HT has not seen the memo itself and relies on the Guardian's account. "Should the public come to believe that the scientific issues are settled, their views about global warming will change accordingly," Luntz wrote, as quoted by the Guardian. "Therefore, you need to continue to make the lack of scientific certainty a primary issue in the debate." According to the Guardian, the memo urged Republicans to "emphasise the importance of 'acting only with all the facts in hand'." In July 2019, Luntz told a Senate Democratic panel on climate, as reported by Anthony Adragna on Yahoo News, "I was wrong in 2001," dating the work a year earlier than the Guardian had. He went on: "Just stop using something that I wrote 18 years ago because it's not accurate today."

The memo concerns scientific certainty in general rather than one measured effect, but its advice matches the weaponized form: keep the uncertainty in the foreground. HT has not found a documented case from the political left.

The weaponized form uses the same words as the honest one. The difference lies in what the person does with the answer.

The honest use asks how big an effect is in order to decide; the weaponized use asks in order to postpone deciding, and no answer is ever precise enough. The honest use accepts a range ("somewhere between modest and large") as enough to act on when the cost of waiting is high, while the weaponized use treats any range as proof of ignorance. The honest use weighs the cost of acting on an imprecise estimate against the cost of not acting; the weaponized use counts only the first. And the honest user demands effect sizes for findings that help their side as readily as for findings that hurt it.

Run it on your own side first

Try these questions:

  • Think of a study or statistic you cite on an issue you care about. Do you know how large the effect is in plain units, or only that it was "significant" or "linked"?
  • Think of a harm your side tends to call "not proven." Are you waiting for a precise number that would never satisfy you, while accepting imprecise numbers on the issues where you already agree?
  • When you last shared a finding that alarmed you, did you know for whom the effect was large, or did you assume it applied to everyone?

Grammars it works against

A grammar is a recurring pattern of argument that turns up across many different subjects. This tool is a first-line defense against G7.

  • G7: Moral Panic and the Folk Devil. In this grammar, public alarm about a group or practice comes loose from the actual scale of harm. Size is the question both poles skip. The people raising the alarm treat any real effect as proof of a crisis, and the people calling "moral panic" treat the panic's excesses as proof the harm is not real. The Effect-Size Reflex asks both how big the harm is, for whom, and compared with what, which can deflate an inflated scare and can equally show that a dismissed one was serious.

Articles that use it