A1: The Invisible-Cases Question

The buildings that are still standing

Imagine you are walking through the old center of a town on a weekend afternoon. The stone church is beautiful. The row of houses along the square has stood for two hundred years. Then you turn a corner onto a street of newer buildings, plain and already a little shabby, and a thought arrives fully formed: they really don't build things the way they used to.

It feels like an observation. You are looking at the evidence with your own eyes. But think about what you were shown. The old buildings in front of you are the ones that lasted. The cheap, badly built houses of two hundred years ago burned, sagged, or were torn down long ago, and nobody kept them for you to compare. Time has already done the sorting. You are comparing the best survivors of one era with an unsorted sample of another, and the verdict was decided by the sorting, not by the builders.

Most people would not stop on that corner to ask what had been removed. That is exactly why this tool exists.

The move

The Invisible-Cases Question is a tool, which on this site means a mental operation you can run on a claim in real time, before you believe it or pass it on. It starts from one plain fact: every collection of evidence that reaches you passed through a filter on the way, and that filter usually removed the cases that would change your conclusion.

Running it takes four steps.

1. Notice that you are about to generalize from a set of cases: success stories, testimonials, survivors, viral examples, the people who wrote in.
2. Ask how these particular cases reached you. Who chose them, what made them visible, and what would have kept a case out?
3. Picture the cases the filter removed. What would they look like, and how many might there be?
4. Ask whether your conclusion would survive if those cases were put back. If you cannot tell, hold the conclusion more loosely until you can find out.

The second step is the heart of it. The question is not "is this evidence real?" It usually is. The question is "real, and selected by what?"

Why it works

Most of the errors this site deals with happen in the reasoning: a bad inference, a slippery word, a convenient comparison. This tool belongs to the family the site calls Bias Literacy, which deals with errors that happen before reasoning starts. A filtered sample can be reasoned about flawlessly and still produce a wrong answer, because the mistake was made before you saw the first case. Careful thinking about a sample cannot correct for cases that were never in it.

Filters are everywhere, and most are innocent. Time removes the shoddy buildings. Businesses that failed do not give talks about their methods. People who tried a remedy and felt nothing rarely write a testimonial. The outrages that travel furthest online are the ones that provoked the strongest reactions, not the ones that best represent what usually happens. None of this requires anyone to be lying. The filter does its work quietly, and what it leaves behind looks like a fair picture of the world.

Researchers in several fields have given names to versions of this problem, among them survivorship bias and selection bias.

Source still being found. a reliable account of where the terms "survivorship bias" and "selection bias" come from and how statisticians define them, from a work actually read

Source still being found. whether the frequently told story of statisticians studying damage on returning aircraft is accurately documented, and if so, a primary or scholarly source for it; leave it out if it cannot be traced

Using it well

The legitimate use is asking what filtered a set of success stories, testimonials, or viral outrages before generalizing from them.

Suppose a friend sends you a long thread in which dozens of people describe quitting a stable job to start a business and say it was the best decision they ever made. The stories are vivid and probably true. Run the question. Who posts a thread about how their leap worked out? Mostly the people for whom it worked. Those who quietly went back to salaried work, or are still paying off the loan, have little reason to post. The thread tells you that some people who made the leap are glad they did. It cannot tell you what share of people who made the leap are glad, which is the thing you actually wanted to know before handing in your notice.

Notice what the tool did not do. It did not prove the stories false or the advice bad. It identified the specific filter (who chooses to post), described the missing cases (people whose leap failed), and sent you looking for evidence that the filter would not have shaped, such as figures on how many new businesses survive their first few years. A good use of the tool ends with a better question, not with a dismissal.

Example still being found. a real, quoted, dated public instance where someone asked what filtered a set of success stories, testimonials or viral cases and the answer changed the conclusion; ideally one where the asker's own side benefited from the original, unfiltered impression

When it becomes a weapon

The weaponized form is invoking "selection effects" only against evidence you don't like, while your own side's anecdotes go unexamined.

This version is hard to spot because each individual use sounds rigorous. Someone shares a set of stories that cut against your view, and you point out, correctly, that the stories were selected. Then someone on your side shares a set of stories just as selected, and you pass it along without a pause. Each objection, taken alone, was sound. Taken together, they amount to a filter of your own, one that removes inconvenient evidence and keeps the rest.

There is a harsher version too. "That's just a biased sample" can be used to wave away any testimony at all, including the testimony of people who were really harmed. A sample being filtered does not make it false. It means the sample cannot, on its own, tell you how common something is. People who use the tool as a weapon tend to slide from the first claim to the second.

A few signs help tell the honest use from the weaponized one:

• The honest use names a specific filter. The weaponized use gestures at "bias" in general.
• The honest use says what the missing cases would look like and where they might be found. The weaponized use stops once the evidence has been discredited.
• The honest use treats a filtered sample as incomplete. The weaponized use treats it as worthless.
• The honest user can show you the last time they applied the question to evidence they liked.

Example still being found. a real, quoted, dated instance of "selection bias", "survivorship bias" or "cherry-picking" being invoked against one side's evidence while comparable evidence on the speaker's own side went unexamined; find examples from both the political left and the political right

Run it on your own side first

This instruction comes with every tool on this site, and it is the actual test of whether the tool has been understood. Try these:

• Think of the stories you find most convincing on an issue you care about. Who collected them, and who had a reason not to tell theirs?
• Think of the last time you shared an outrage about the other side. How did it reach you, and how many quieter, more ordinary cases of the same kind would it have taken to make it unremarkable?
• When you picture the past, the place you grew up, or the way things used to be, which parts of it have you forgotten, and which did nobody bother to keep?

If the answers are uncomfortable, the tool is working.

Grammars it works against

A grammar, on this site, is a recurring pattern of argument that turns up across many different subjects. This tool is one of the first ones to reach for against two of them.

• G5: Decline and Golden Age. When a curated version of the past is held up as the standard the present is failing to meet, this tool asks what did the curating. The past that survives in memory and in buildings tends to be its best part. The question works in both directions: it catches the declinist who forgets what the past was really like, and it equally catches the progress triumphalist who dismisses every claim of decline, when some decline is measurable and the evidence for it may itself have been filtered out of view.
• G10: Victimhood as Standing. When harm is treated as a credential, the grievances that become visible are a selected set, and this tool asks what selected them. Used honestly, it checks both those who inflate grievance and those who deny it by reflex, since the injuries that never get a hearing are invisible cases too.

Related tools

• A2: The Denominator Demand asks "out of how many?" A1 asks which cases were left out before you ever saw the number, and the two are often run together.
• A3: The Newsfeed Inversion deals with one especially powerful filter: how often something appears in a feed reflects how well it performs, not how common it is.
• B4: The Reference-Class Question asks what a case should be compared with. A1 often reveals that the comparison was already rigged by a filter.
• A4: The Mirror Check is the guard against A1's weaponized form: did you ask what filtered this evidence because the question applied, or because you disliked the conclusion?