B4: The Reference-Class Question

Two crowds on the same road

Imagine a crowd of a few thousand people sitting down across a main road at rush hour. Traffic stops for two hours. Commuters miss work, and an ambulance has to find another route. The protesters are there for a cause you care about deeply.

What is this a case of? For most people the answer arrives before the question does: civil disobedience, part of an honorable line of people who broke small rules to expose large wrongs.

Now suppose the same crowd, on the same road, for the same two hours, is there for a cause you find wrong or even dangerous. The answer arrives just as fast, and it is different: a mob holding ordinary people hostage.

Nothing about the road, the crowd or the ambulance changed. What changed was the comparison you reached for, and you probably did not notice reaching for it. That silent choice did most of your thinking for you.

The move

The Reference-Class Question is a tool, which on Hardly Thinking means a mental operation you can run on a claim in real time, before you believe it or pass it on. Its tagline gives the method: before you judge something, ask what it is a case of, and notice that the answer is a choice.

A reference class is the group of earlier cases you compare a new one with. Nobody judges anything new from nothing. Is a new technology dangerous? Is a scandal serious? Is a protest legitimate? Each answer depends on what you set it beside. Usually the comparison goes unstated, and by the time a verdict reaches you, the important choice has been made.

Running the tool takes four steps.

1. Notice the verdict you have reached, or the one a writer is offering you.
2. Ask what comparison it depends on. What earlier cases is this being treated as one of?
3. Name two or three other plausible comparison classes, ones a thoughtful person on another side might choose.
4. Rerun the verdict against each, and see how much it moves.

If the verdict barely moves, you can hold it with more confidence. If it swings wildly, you have learned that the real argument is about the comparison, not the case.

Why it works

Judging anything new means silently importing a comparison class, and that import is where the real reasoning happens. The tool's job is to make the import visible.

It belongs to the family Hardly Thinking calls Inference Discipline, which deals with errors in the step from evidence to conclusion. The facts about a case can be accurate and agreed on, and two people can still reach opposite verdicts, because they have placed those same facts in different classes. Arguing over the facts will not settle that. Naming the classes might.

The choice of class is rarely forced. Most cases resemble several kinds of earlier case in some respects and not in others. Choosing which resemblance counts is a judgment, and a stated judgment can be examined and disputed. An unstated one cannot.

The same choice turns up in forecasting. In a 1977 report, the psychologists Daniel Kahneman and Amos Tversky set out a remedy for errors of intuitive prediction that began with choosing a reference class, and they warned: "More often than not, the expert will think of several classes to which the problem could be referred, and a choice among these alternatives will be necessary." Kahneman called the case-by-case approach the "inside view" in a 1993 paper with Dan Lovallo; its opposite is known as the "outside view."

By his own account, the planning scholar Bent Flyvbjerg made the first practical use of the method, which he calls "reference class forecasting," in work that led the British government to adopt it for large transport projects in 2004. In a 2006 paper he wrote that the class "must be broad enough to be statistically meaningful but narrow enough to be truly comparable with the specific project." Welton Chang, Philip Tetlock and two colleagues at the University of Pennsylvania studied forecasting tournaments sponsored by the U.S. intelligence community. They reported in 2016 that less than an hour of training, which included "using comparison classes and base rates," improved forecasters' accuracy scores by 6 to 11 percent compared with untrained forecasters.

Philosophers call the difficulty the "reference class problem." The logician John Venn described it in the first edition of The Logic of Chance in 1866: a man named John Smith, "a consumptive man say, and a native of a northern climate," belongs to many classes at once. The philosopher Hans Reichenbach wrote in The Theory of Probability (1949): "An individual thing or event may be incorporated in many reference classes, from which different probabilities will result. This ambiguity has been called the problem of the reference class." The philosopher Alan Hájek argued in 2007 that versions of most of the leading theories of probability face the problem, not only the frequency theory Venn and Reichenbach held.

Using it well

The legitimate use is naming two or three plausible comparison classes for a new risk or scandal and noticing how much the verdict shifts between them.

Prices in the United States were rising in the summer of 2021, as the economy reopened after the pandemic. On July 6, the White House Council of Economic Advisers published a blog post by its chair, Cecilia Rouse, with Jeffery Zhang and Ernie Tedeschi. It set the moment beside each period since World War II in which inflation had reached 5 percent. The three most recent episodes "were largely a function of oil shocks," which, the authors wrote, had not been the main driver this time. The period right after World War II, they wrote, "potentially provides the most relevant case study," because its inflation came from "the elimination of price controls, supply shortages, and pent-up demand." They named a "substantial difference" as well: wartime price controls, which the pandemic did not have, had held prices down, and prices climbed when they were lifted.

The post shows the tool's first steps in public. It names several classes, states the features that decide between them and says where the best one differs, so a reader who disagrees can argue about the comparison rather than trade verdicts. The authors were President Joe Biden's own advisers, and the class they preferred pointed to a reassuring answer for his administration: the post-war period "is likely a better comparison for the current economic situation than the 1970s and suggests that inflation could quickly decline once supply chains are fully online and pent-up demand levels off." They did not say what the 1970s, or any other class, would have predicted, and their choice is open to the charge of class-shopping, the tool's weaponized form. What the post does is put that choice where anyone can check it. Source still being found. how inflation ran after the post (Bureau of Labor Statistics consumer price index, 2022)

When it becomes a weapon

The weaponized form is class-shopping: picking the comparison that flatters your preferred conclusion and presenting it as the obvious one.

The breach of the Capitol on January 6, 2021, has been placed in rival classes ever since. At a House Oversight and Reform Committee hearing on May 12, 2021, Rep. Andrew Clyde, a Georgia Republican, said the day "was not an insurrection." He said the footage of Statuary Hall showed people "staying between the stanchions and ropes taking videos and pictures." "If you didn't know the TV footage was a video from January 6," he said, "you would actually think it was a normal tourist visit." Clyde also said he had helped "barricade the door until almost 3 p.m. that day from the mob who tried to enter."

On the first anniversary, Vice President Kamala Harris, a Democrat, chose the opposite class. "Certain dates echo throughout history," she said in a speech at the Capitol, naming dates when "our democracy came under assault": "December 7th, 1941. September 11th, 2001. And January 6th, 2021." The feature she named was a place in "our collective memory."

Clyde tested the word "insurrection" against its definition, but the alternative he put to a witness, "a riot with vandalism, similar to what we saw last summer," set the day beside the protests of 2020. His tourist comparison rests on a surface resemblance, and his own account of barricading a door is the feature it leaves out. Harris offered one class as though no other existed. Both earlier dates were attacks from abroad. Harris did not say what set those dates apart from January 6.

From outside, the honest use and class-shopping can look the same; the difference lies in how the comparison was chosen. The honest use names more than one class; class-shopping offers one, as though no other existed. The honest use explains why the chosen class shares the features that matter; class-shopping relies on a surface resemblance. The honest user would accept the same kind of comparison if it cut against them; the class-shopper picks a different standard each time and always lands on the same side. The honest use also allows that a case may be mixed.

Run it on your own side first

Every tool on HT carries this instruction, because it tests whether the tool has been understood.

• Think of the comparison you reach for most often on an issue you care about, the "this is just like…" that settles it for you. When did you last consider a different comparison, and what verdict would it give?
• Think of an event your side defended and a similar event your side condemned. Did you place them in different classes? Would you accept the other side's placing them the other way round?
• When a writer you trust says something "is a case of" something else, do you check the comparison, or only when a writer you distrust says it?

If the road-blocking protest came out differently for you depending on the cause, you have already found your first case.

Grammars it works against

A grammar, in HT's terms, is a recurring pattern of argument that turns up across many different subjects. This tool works against two of them.

• G5: Decline and Golden Age. Every claim that things used to be better depends on a chosen past: which decade, which place, whose life. The tool asks whether that past is a fair sample or a curated highlight. It questions both the declinist who compares today with a remembered peak and the progress triumphalist who measures today only against a worse past, reads history as a straight march toward the present, and dismisses decline that is real and measurable.
• G12: Fixed Pie or Growing Pie. Whether a situation feels zero-sum depends on what you take it to be a case of. Treat a new job market as a case of dividing a fixed stock and every newcomer looks like a loss; treat it as a case of trade and every newcomer looks like a gain. The tool asks which parts of the situation resemble a fixed stock, such as land or rank, and which resemble exchange, so that neither pole gets to pick the flattering class for the whole.

Related tools

• A2: The Denominator Demand asks what a number is out of. Choosing a denominator is choosing a reference class, and both can be shopped for.
• A1: The Invisible-Cases Question asks what filtered the cases you can see. A reference class built only from the cases that survived to be remembered will mislead.
• F1: The Crux Finder looks for the specific question a disagreement depends on. Often it is which comparison class applies.
• A4: The Mirror Check asks whether you scrutinized a claim or simply liked its conclusion. It keeps this tool from sliding into class-shopping.

Where HT uses it

HT's article on the 2026 backlash against AI data centers uses this tool: Do Data Centers Raise Your Electric Bill?