C4: The Prediction Ledger

This is one of the thirty thinking tools identified and explored on Hardly Thinking. Explore all the tools →

Write it down, date it, grade it later.

The warning you remember getting right

Imagine a debate from five years ago that you followed closely. It might have been about a new technology, a change in the law, a trend among teenagers, an economic policy. The people you trusted warned that it would lead somewhere bad, or promised that the worries about it were overblown. Today you have a firm sense that your side called it.

Now suppose someone asked you what, exactly, your side predicted. Not the mood of it, but the claim: what would happen, by when, and how you would know. Most of us could not answer. We remember the general direction of our side's warnings and the specific moments that seemed to bear them out. The predictions that quietly failed to arrive have mostly slipped away, and so have the ones the other side got right. The feeling of having been right survives. The record that would test it was never kept.

The move

The Prediction Ledger is a tool, which on this site means a mental operation you can run on a claim in real time, before you believe it or pass it on. Its content is in the tagline: when someone makes a prediction, write it down, date it, and come back later to grade it.

A prediction, for this purpose, is any claim about what will happen. Public arguments are full of them, though they are seldom labeled as forecasts. "This law will empty the cities." "This fear will look silly in ten years." "If we allow this, that will follow." Each is a claim that the future can check, if anyone bothers to.

Running the tool takes four steps.

Notice the forecast hiding in the argument, and write it down in the speaker's own words where you can.
Make it specific enough to grade. What, exactly, would happen? By when? What would you see if it came true, and what would you see if it did not?
Date it, and note who made it and how confident they sounded.
Come back when the time is up and grade it honestly: right, wrong, or partly right, with the same rules for every speaker, including the ones you agree with.

The second step does most of the work. Many forecasts cannot be graded as stated, because they name no date and no observable outcome. Noticing that is itself useful. A warning that could never turn out wrong is not really telling you anything about the future.

One note on how the site itself uses this tool. Hardly Thinking makes no forecasts of its own; most of what it covers is not shaped as a prediction, and the site holds itself to account through reader review instead. The Prediction Ledger is used to examine the predictions made by the sources an article analyzes.

Why it works

The reasoning fits in six words: vibes have no memory; ledgers do. A general impression of who has been right is formed by whatever was memorable, and what is memorable is mostly what confirmed what we already believed. A written, dated record does not edit itself that way. It holds the misses as firmly as the hits.

This tool belongs to the family the site calls Testing and Calibration, which is about proportioning belief to evidence. A track record is one of the best kinds of evidence about a source. Someone whose confident warnings have repeatedly come true has earned more weight for the next one, and someone whose confident warnings have repeatedly failed has earned less. Without a ledger, that weighing is done by memory and loyalty, which is to say it is not really done.

Two bodies of research bear on this: work by psychologists on how people remember their own past predictions, and work by researchers who record and score forecasts. What that research actually found will be summarized here once the works themselves have been read.

Source still being found. research on hindsight bias, the tendency to remember one's past predictions as closer to what actually happened than they were, from a work actually read

Source still being found. research on the accuracy of expert and pundit forecasts when they are recorded and scored, and on whether keeping score improves forecasting, from a work actually read

Using it well

The legitimate use is logging a specific, dated, graded prediction and returning to check it honestly.

Suppose a town is arguing about a new late-night entertainment district. One camp says it will bring a crime wave within two years. The other says it will revive the town center and the fears are a panic. Run the tool. Write both forecasts down. Make them gradable: suppose "crime wave" is pinned to "reported assaults in the district at least double within two years," and "revive the town center" is pinned to "fewer empty storefronts on the main street in two years." Date both, and note who said each.

Two years later, suppose assaults rose a little but did not double, and the empty storefronts fell by a few. Graded honestly, neither camp got what it predicted. The alarm was overstated, but the dismissal of it was not fully borne out either, since the rise in assaults was real. Without the ledger, each camp would likely remember having been right. With it, the town has something better than a verdict: a sense of how far to trust each camp's next forecast, and a real, modest problem to deal with.

A good use of the tool sometimes ends with the discovery that the side you doubted called it correctly. When that happens, the tool has done its job.

Example still being found. a real, quoted, dated public prediction (ideally a warning and a counter-warning in the same debate) that can now be graded against what happened, from a debate the bible lists under G7 or G5, such as past media or technology scares, crime forecasts, or predictions of civilizational or demographic collapse; include at least one prediction from a source the reader would be inclined to trust

When it becomes a weapon

The weaponized form is selective grading: judging opponents on outcomes and allies on intentions.

Suppose two columnists on opposite sides both made confident forecasts that failed. A reader who has learned this tool goes back and finds the opposing columnist's failed prediction, quotes it in full, and declares that columnist discredited. When the same reader's own columnist is shown to have missed, the grading changes. The prediction "was really a warning meant to prevent the outcome," or "was right in spirit," or "would have come true if not for events no one could foresee." Each excuse may sometimes be fair. Applied to one side only, they turn a record-keeping tool into a scorekeeping one, where the score is decided before the game.

This deserves as much attention as the honest use, because the weaponized form looks like rigor. It quotes, it dates, it checks. What gives it away is that the rules change depending on who is being graded.

A few signs help tell the two apart:

The honest ledger records forecasts from every side before the outcome is known. The weaponized one goes looking for an opponent's prediction after it has failed.
The honest ledger grades a miss as a miss whoever made it. The weaponized one accepts excuses, such as good intentions or a warning that worked by being heeded, only from allies.
The honest ledger counts the hits of the other side as well as its misses.

Example still being found. a real, quoted, dated instance of selective grading, in which a writer held an opponent to a failed prediction while excusing a comparable miss by their own side; find examples from both the political left and the political right

Run it on your own side first

Every tool on this site comes with this instruction, and it is the real test of whether the tool has been understood. Try these:

Name one specific prediction made by a writer or public figure you trust that did not come true. If none comes to mind, is that because there are none, or because you never kept track?
Think of a warning from the other side that you dismissed years ago. Do you know whether it came true, or only that you stopped hearing about it?
When your side's forecast failed, what reason did you accept for it? Would you accept the same reason from the other side?

If you can name failed predictions only for people you disagree with, you have been keeping half a ledger.

Grammars it works against

A grammar, on this site, is a recurring pattern of argument that turns up across many different subjects. This tool is one of the first to reach for against one of them and gives support against another.

G7: Moral Panic and the Folk Devil. Here it is a first-line tool. In this grammar, alarm about a group or practice comes loose from the actual scale of harm, and the counter-panic can come just as loose in the other direction. Panics usually fade without anyone auditing their original claims. Every panic makes a forecast, and so does every dismissal of one; the ledger turns that fading into an audit of both.
G5: Decline and Golden Age. Here it gives support. Stories of decline predict further collapse, and stories of steady progress predict further improvement. Writing those forecasts down and grading them later tests both, and separates measurable decline from a mood of decline.

C1: The Falsifier Question asks what would change your mind, written down in advance. It shares C4's habit of committing before the outcome is known; C1 does it for beliefs, C4 for forecasts.
B1: The Effect-Size Reflex is the other first-line tool against G7. Where C4 checks later whether a warning came true, B1 checks now how big the harm is.
A4: The Mirror Check asks whether you scrutinized a claim or just liked its conclusion. It is the direct guard against C4's weaponized form, selective grading.
A1: The Invisible-Cases Question asks what you are not seeing and what filtered it out. Forgotten failed predictions are exactly the kind of case that memory filters out.