C1: The Falsifier Question

The program that was always about to work

Imagine your city launches a program you argued for. Perhaps it is meant to cut traffic deaths, or reduce homelessness, or raise reading scores. You argued for it in public and told a skeptical relative it would work.

A year later the figures come in, and they are flat. Nothing much has changed. Within minutes, reasons arrive. It is too early to judge. The program was underfunded. Things would have been worse without it. Something else happened that year and swamped the effect. Each might be true, which is what makes the moment so comfortable.

Now imagine the same flat figures for a program you argued against. You would probably call it a failure, and you would probably be annoyed by anyone who said it was too early to tell.

The trouble began a year earlier, when you never said what result would count against the program. With nothing written down, any result could be read as support, and you were the one doing the reading. It is easy to go a whole year without asking yourself that question. That is why this tool exists.

The move

The Falsifier Question is a tool, which on this site means a mental operation you can run on a claim in real time, before you believe it or pass it on. It rests on one idea: a belief that no observation could ever count against isn't a belief about the world. If every possible outcome fits it, it is not telling you anything about which outcome to expect.

Running it takes four steps.

1. Pick a factual belief you hold with some confidence: that a policy will work, that a group is behind an event, that a trend is real.
2. Ask what you would expect to see if the belief were wrong. Make it specific and checkable: a number, an event, a document, a result.
3. Write it down, with a date, before you know whether it will happen.
4. When the evidence arrives, compare it with what you wrote, not with what you now feel like counting.

The third step is the one people skip, and it carries most of the weight. A falsifier named after the fact is easy to tailor, because by then you know which answer you want.

Why it works

This tool belongs to the family the site calls Testing and Calibration, which is about proportioning belief to evidence: holding a belief as firmly as the evidence supports, and no more firmly. You cannot proportion a belief to evidence if you have not decided what evidence would bear on it. Without a falsifier, evidence does not move the belief. It only decorates it.

Writing the falsifier in advance matters because of the order in which most of us reason. Once a result is known, the mind is very good at finding a story that makes the result fit what it already believed. What gives it away is that such a story is always available, whichever way the result goes. A falsifier written down beforehand takes that freedom away. You agreed to the terms before you knew how the game would end.

The idea that a claim about the world should say what would show it false has a long history in the philosophy of science.

Source still being found. an accurate account, from a work actually read, of Karl Popper's falsifiability criterion and what he did and did not claim for it, including the main criticisms of it

Source still being found. research on hindsight bias or post-hoc rationalization showing that people reinterpret evidence to fit prior beliefs once outcomes are known, from a work actually read

There is a limit worth stating plainly. The tool is for factual beliefs, beliefs about what is or will be the case. Many deeply held positions also rest on values, such as how much weight to give liberty against safety. A value is not the kind of thing an observation can refute, and that is not a defect. Part of running this tool honestly is separating the factual parts of a position, which should have falsifiers, from the value parts, which need a different kind of argument.

Using it well

The legitimate use is stating a specific, checkable observation that would count against a confident position before the test happens.

Suppose you believe a new rule requiring calorie labels on restaurant menus will lead people to eat less. Running the tool, you write: "If, two years after the rule takes effect, careful studies of what people actually order show no meaningful drop in calories per meal, I will stop believing the labels work as intended." It commits you to a kind of evidence (what people order, not what they say). It sets a time frame, so "too early to tell" cannot run forever. And it gives anyone who disagrees with you a fair way to hold you to account.

Notice what the tool did not do. It did not make you less confident, or tell you whether the labels will work. It turned a confident feeling into a claim that can win or lose. If the evidence comes in as you predicted, your confidence is now earned. If it does not, you have learned something, and you have learned it at the cheapest possible moment.

Example still being found. a real, quoted, dated instance of a public figure, researcher or institution stating in advance what result would count against their own confident position, ideally followed by the result and how they responded to it

When it becomes a weapon

The weaponized form is demanding falsifiers from opponents while offering none of your own.

"What evidence would change your mind?" is a fair question, and asking it of someone else can expose a belief that has quietly become immune to evidence. The weapon is not the question. It is the one-way traffic. A person who asks it of every opponent and never answers it about their own positions is not testing beliefs. They are using a respected method to put the other side on trial while their own side is never called.

There is a subtler version too. Someone names a falsifier, and the questioner moves the bar: that evidence would not count, the study was flawed, the sample was wrong. Some of those objections may be fair. But if no falsifier the other person offers is ever accepted, then the questioner's demand has itself become unfalsifiable, which is the very failure the tool was meant to catch.

A few signs help tell the honest use from the weaponized one:

• The honest user can tell you, without much hesitation, what would change their own mind on the same question.
• The honest user accepts a reasonable falsifier when one is offered, even if they would have chosen a different one.
• The honest user asks for falsifiers about factual claims, not about values dressed up as facts.
• The honest user asks the question of their allies as well as their opponents.

Example still being found. a real, quoted, dated instance of someone demanding that opponents say what would change their minds while declining to state their own falsifier on the same question; find examples from both the political left and the political right

Run it on your own side first

This instruction comes with every tool on this site, and it is the actual test of whether the tool has been understood. Try these:

• Take the political belief you are most confident about. What specific event or figure, if it came to pass next year, would make you less sure? If you cannot name one, what does that tell you?
• Think of a prediction your side made that did not come true. When it failed, did you treat it as evidence, or did reasons arrive to explain it away?
• Imagine someone you disagree with asks you the question in public. Would you answer it, or turn it back on them?

If the answers are uncomfortable, the tool is working.

Grammars it works against

A grammar, on this site, is a recurring pattern of argument that turns up across many different subjects. This tool is one of the first ones to reach for against one of them.

• G6: The Hidden Hand. This grammar explains events by a coordinated actor working in secret, and it is dangerous because it is sometimes right. One pole produces a belief that cannot lose: every missing piece of evidence is read as proof of a cover-up. Asking "what would show there was no plot?" exposes that. But the question applies to both sides of this grammar. Someone who dismisses every claim of coordination as "conspiracy theory" should also be able to say what evidence would show that a particular coordination was real, since some of it has been, and the honest answer should not be "nothing."

Related tools

• C4: The Prediction Ledger is C1's partner: once you have written down what you expect, the ledger is where you date it and go back to grade it.
• F1: The Crux Finder asks what would need to be true for each side to switch positions, which is C1 applied to both parties of a disagreement at once.
• D4: The JAQ Detector watches what happens when an answer arrives to a question, which is often the moment that shows whether a falsifier was ever meant seriously.
• A4: The Mirror Check is the guard against C1's weaponized form: are you asking for falsifiers from everyone, or only from the people whose conclusions you dislike?