C3: The Consensus Audit

The experts you have never met

Imagine you are in an argument about something you care about, and you say, "The experts agree on this." You believe it. You have read it many times, in sources you trust.

Now suppose the other person asks, calmly, which experts. In what field? How strongly do they agree, and about all of what you said or only part? Most of us could not answer. We are not lying; we are repeating what someone we trust told us the experts said.

The same thing can happen in reverse. If the other person had named one professor who disagrees, you might have felt the ground shift or a surge of contempt. Both reactions skip the same questions: what field the dissenter works in, and how many stand with them.

The move

On Hardly Thinking, a tool is a mental operation you can run on a claim in real time, before you believe it or pass it on. The Consensus Audit starts from an admission: you cannot check the evidence for most of what you believe, so you are going to rely on other people's judgment whether you like it or not. The tool does not ask whether to defer. It asks how well you are deferring.

Running it takes five steps.

1. State exactly what is supposed to be agreed. Write it in one sentence, and notice if it is really two or three claims.
2. Ask whether the people said to agree work in the field that actually studies this question, or in a neighboring one that only sounds relevant.
3. Ask how strong the agreement is. Is it near-unanimous, a solid majority, a working assumption, or the statement of a single organization?
4. Ask what the dissent looks like. Who disagrees, from which field, about which part of the claim, and how many of them are there?
5. Set your confidence to match what you found, and say it in those terms, for instance "settled at the core, open at the edges."

The fifth step matters most: the audit's product is a level of confidence, not a verdict on who is right.

Why it works

The audit removes a false choice that runs through many public arguments, the choice between "trust the experts" and "think for yourself." The difference between careful and careless people is not whether they defer but to whom, about what, and how much.

The philosopher John Hardwig argued that deference is unavoidable and often rational in "Epistemic Dependence," a 1985 paper in The Journal of Philosophy. "I believe too much ... intellect is too small and life too short," he wrote. He concluded that "rationality sometimes consists in refusing to think for oneself." He doubted, though, that laypeople can tell good experts from bad: by asking the right questions, he wrote, a layperson "might be able to spot a few quacks, phonies, or incompetents, but only the more obvious ones." The audit is an answer to that doubt, not part of his argument: it asks who agrees, in what field and how strongly, which a layperson can often check even when the evidence itself is beyond them.

The Consensus Audit belongs to the family Hardly Thinking calls Testing and Calibration. When you cannot weigh the evidence yourself, the strength and relevance of expert agreement is the evidence you have: a broad consensus inside the right field deserves more weight than a thin majority, an adjacent field, or one organization speaking for a discipline.

Surveys of researchers show why the field matters. A survey of 3,146 Earth scientists, published in 2009, asked whether "human activity is a significant contributing factor in changing mean global temperatures." According to a 2016 paper by John Cook and colleagues, several of whom work with Skeptical Science, a website that rebuts climate skeptics, 82 percent of all respondents agreed, and 97.4 percent of the 77 respondents in "the subset with greatest expertise in climate science." In the same journal, Richard Tol, a University of Sussex economist who has criticized Cook's work, named a second limit: "Different studies may have different objects of consensus." One survey he cited asked four related questions and got four levels of agreement, from 71 to 96 percent. Each figure measures agreement with the question that was asked, and only that question.

A related failure runs through the reader rather than the expert. The novelist Michael Crichton named it Gell-Mann amnesia in a talk titled "Why Speculate?" in La Jolla, California, on April 26, 2002. You read a newspaper story on a subject you know well and find it full of errors, he said, then turn to other news "as if the rest of the newspaper was somehow more accurate about far-off Palestine than it was about the story you just read. You turn the page, and forget what you know." The audit asks you to apply what you know about a source's errors in your own field to what it tells you about other fields.

Using it well

In a January 30, 2020, comment in Nature, two climate researchers ran the audit on their own side. Zeke Hausfather of the Breakthrough Institute, an environmental policy think tank, and Glen Peters of the CICERO Center for International Climate Research in Oslo took up RCP8.5, the most extreme of four warming scenarios used in the 2014 report of the United Nations' climate panel. The scenario "was intended to explore an unlikely high-risk future," they wrote, but some experts, policymakers and the media had used it "as something else entirely: as a likely 'business as usual' outcome." They also pointed to a problem of field: "many climate researchers are not familiar with the details of these scenarios in the energy-modelling literature."

They did not dispute the core. An assessment of current policies, they wrote, suggests the world is "on course for around 3 °C of warming above pre-industrial levels by the end of the century — still a catastrophic outcome, but a long way from 5 °C." They also reported the dissent: "Some researchers argue that RCP8.5 could be more likely than was originally proposed." The result was neither "the science is settled" nor "the experts are hiding something." They left the core standing and marked one edge as open: how much warming the world is on course for. An audit that confirms a consensus has done its job as well as one that weakens it.

When it becomes a weapon

The first weaponized form is "the science is settled" stretched over questions that are genuinely unsettled. A real consensus on a core finding is used to cover predictions, policies or value judgments the experts never agreed on, and anyone questioning the uncovered part is treated as denying the core. When the stretched part proves wrong, people conclude the core was wrong too.

The second is trophy-dissident hunting: finding one credentialed contrarian and declaring the consensus broken. A single professor, a long petition or a famous name from outside the field is presented as proof that the experts are divided, when the field itself is not.

On May 16, 2013, a post on President Barack Obama's Twitter account read: "Ninety-seven percent of scientists agree: #climate change is real, man-made and dangerous." The figure came from a study by John Cook and colleagues published the day before. According to the later papers by Cook's team and by Tol, it counted research papers, not scientists, and measured agreement that humans are causing recent warming, not that the warming is dangerous. On January 14, 2025, Rep. Pete Stauber, a Minnesota Republican, voted for a bill that his office said would "prohibit biological men from participating in women's sports." His statement read: "It is a scientific fact that men and women have biological differences that make competing on the same sports teams unfair and potentially dangerous." That the sexes differ physically is established. Whether the differences make competition "unfair" is partly a judgment about how sport should be organized, and how much risk of injury would justify exclusion is another. Each statement lends a finding's authority to a conclusion the finding does not reach.

The Global Warming Petition Project, an advocacy effort begun in 1998 to dispute the climate consensus, has listed 31,487 signers on its website since at least 2010, and still did in October 2026. It says that "if there is a consensus among American scientists, it is in opposition to the human-caused global warming hypothesis." The audit asks how many signers work in the relevant field. The project's own breakdown, by academic degree rather than current work, counts 39 signers trained in climatology, 112 in atmospheric science and 343 in meteorology.

In 2012 the campaign for Proposition 37, a California ballot measure to require labels on genetically modified food, held a press conference with Gilles-Éric Séralini, a researcher at the University of Caen, to announce his study linking genetically modified corn to tumors in rats. "GM foods have been evaluated in an extremely poor and lax way with much less analysis than we have done," Séralini said, according to Steve Baragona of Voice of America. Labeling advocates "have used the French study to bolster their argument," Baragona wrote. That October, he reported, the French national food safety agency had joined six scientific academies "in concluding the study was too badly done to support its conclusions." HT found no campaign statement saying the scientific consensus was broken; the example rests on the campaign's promotion of a single study. These two cases do not show whether the move is more common on one political side than the other.

A few signs help tell the honest audit from the weapon:

• The honest audit splits the claim into its settled and unsettled parts. The weapon treats it as one block, whether to defend or to attack.
• The honest audit counts dissent and asks where it comes from. The weapon either quotes a single dissenter or refuses to look at any.
• The honest audit can end in more confidence than it started with. The weapon ends at the conclusion it started with.

Run it on your own side first

Every tool on HT carries this instruction, because applying a tool to your own side is the test of whether it has been understood.

• Think of an issue where you often say "the experts agree." Could you name the field, describe how strong the agreement is, and say which part of your claim it actually covers?
• Think of an expert you admire who speaks outside their own field. Do you give them the same deference there that you give them at home?
• Think of the last time a lone dissenter's view pleased you. Did you check how many in their field stood with them, or did you stop looking once you liked the answer?

If the first question was hard to answer, the confidence came from the people you trust, not from anything you checked.

Grammars it works against

On HT, a grammar is a recurring pattern of argument that turns up across many subjects. The Consensus Audit is a first-line tool against one of them.

• G11: Contested Epistemic Authority. This grammar is the fight over who gets to be believed, when credentialed experts, lived experience and institutions point in different directions. Its deep question is "when should you defer, to whom, about what?". The Consensus Audit asks it and checks both poles: the credentialists who stretch authority beyond its field and treat "trust the science" as a slogan, and the populists who generalize from a few expert failures to never deferring at all.

Related tools

• C5: The Cynicism Firewall is the other first-line tool against G11. When an audit turns up a real expert failure, C5 keeps you from concluding that nothing can be trusted.
• F2: The Standing/Soundness Split separates who may speak from whether the claim is true. The audit weighs credentials as evidence; F2 stops credentials, or their absence, from settling the question by themselves.
• D1: The Incentive Trace asks who profits from your believing something. It is a useful companion if you still check the claim afterward.
• A4: The Mirror Check, one of four tools that work against every grammar, asks whether you audited a consensus because you doubted it or only because you disliked its conclusion.

In use: The Childhood Vaccine Schedule: Who Counts as an Expert