A4: The Mirror Check

Two studies on the same morning

Imagine two studies land in your feed on the same morning, both about a question you care about. The first finds what you have always suspected. You read the headline, nod, and share it: "Not surprised." The second finds the opposite. This one you open properly. You notice the sample was small, find that one of the authors once worked for an organization you distrust, and conclude that it proves nothing.

Everything you noticed about the second study may be true. But you never looked at the first study's sample size, and you have no idea who funded it. You let one study through the door and frisked the other. From the inside, this does not feel like bias. It feels like good judgment about one study and proper care about the other.

One study, read two ways

That morning is imaginary, but the habit is real, and it is easiest to watch when a single report lands on both sides at once. On July 30, 2018, the Mercatus Center at George Mason University, a libertarian policy center that, as the Associated Press reported, receives funding from the conservative Koch brothers, published an estimate that Senator Bernie Sanders's "Medicare for all" plan would raise federal government spending on health care by $32.6 trillion over ten years.

That day the Speaker of the House, Paul Ryan, a Republican, tweeted the headline number and added: "Even doubling all federal individual and corporate income taxes wouldn't cover this cost. It is just absurd." Both points match the AP's account of what the report found.

Sanders answered in a written statement of two sentences. The first argued from other countries: "If every major country on earth can guarantee health care to all, and achieve better health outcomes, while spending substantially less per capita than we do, it is absurd for anyone to suggest that the United States cannot do the same." The second answered the report itself only with who had paid for it: "This grossly misleading and biased report is the Koch brothers response to the growing support in our country for a 'Medicare for all' program." Also that day, in a video, he cited the same study as an authority: "Let me thank the Koch Brothers of all people for sponsoring a study that shows that Medicare for All would save the American people $2 trillion over a 10-year period."

The $2 trillion was in the report too, but it measured something different from Ryan's figure. The $32.6 trillion was the rise in federal government spending. The $2 trillion, as PolitiFact explained, was a fall in total national health spending, public and private together, and it came from the study's more generous assumptions, such as paying every doctor and hospital at Medicare's lower rates. Under the report's other scenario, in which cost control works less well, national health spending rose by about $3.3 trillion. PolitiFact rated Sanders's claim Half True: he had "cherry-picked the more flattering of two estimates," and his word "would" was "too strong" for one of two scenarios.

Is it a double standard to call a report biased and then cite it? Not by itself. If you believe a source leans against you, a finding from it that favors you is, if anything, more believable, in the way a confession counts for more than a boast. Sanders's tweet that day made exactly that point: "Thank you, Koch brothers, for accidentally making the case for Medicare for All!" PolitiFact described the thanks as cheeky; what it faulted was the choice of scenario. That is where the Mirror Check catches him: the study offered two estimates, and he reported the one he liked as what "would" happen.

In this case, the clearer lapse was on one side. Ryan quoted the report's headline figure as the report and the AP described it, and nothing we have read shows him judging this study more gently than he would a comparable one. The evidence that the habit runs across politics comes from the research below and the cases further down, not from this one.

The move

The Mirror Check is a tool, which on this site means a mental operation you can run on a claim in real time, before you believe it or pass it on. We tend to ask "can I believe this?" of claims we want to be true, and "must I believe this?" of claims we resist. Almost anything can be believed if you are looking for permission, and almost nothing must be believed if you are looking for an exit. Two people can look at the same evidence, one asking each question, and both feel they have been rigorous.

Running the check takes four steps.

1. Notice that you have reached a verdict on a claim, especially a quick one, and especially one that pleased you.
2. Name the checking you actually did. Did you look at the source, the data, the sample, the funding? Or did you read the headline?
3. Imagine the same claim, with the same evidence, pointing toward a conclusion you would dislike. List the questions you would have asked of it.
4. Ask those questions of the claim in front of you, and let the answers move your verdict, whichever way they go.

The third step is the mirror. You are not asking whether the claim is true; you are asking whether you held it to the standard you hold its opposite to.

Why it works

The tool belongs to the family the site calls Bias Literacy, which deals with errors that happen before reasoning starts: the double standard lives in the earlier decision about how hard to look. It is one of five tools this site treats as most important across every subject, because asymmetric scrutiny is the master bias that every other tool can be corrupted to serve; asking for a denominator becomes a weapon the moment you do it to one side only. That is the site's working judgment, not a research finding, and its articles will test it.

The "can I, must I" phrasing is credited by researchers such as Bojana Kuzmanovic and Lionel Rigoux (2017) to the psychologist Thomas Gilovich's 1991 book How We Know What Isn't So, alongside two later papers. They describe people accepting welcome information easily ("Can I believe this?") but applying "an effortful scrutiny" to unwelcome information ("Must I believe this?"). We have not read Gilovich's own passage, so the wording here is theirs. Psychologists call the wider pattern motivated reasoning, and this kind of lopsided checking is sometimes called motivated skepticism, the title of a 1992 paper by Peter Ditto and David Lopez. Those are the terms to search for if you want more.

The research behind the idea is substantial. In 2019 the psychologist Peter Ditto and colleagues (a free copy of their accepted manuscript is on OSF) combined 51 American studies, with more than 18,000 participants, that measured "the tendency to evaluate otherwise identical information more favorably when it supports one's political beliefs or allegiances than when it challenges those beliefs or allegiances." The classic version is a 1979 study by Charles Lord, Lee Ross and Mark Lepper. As Ditto and colleagues summarize it, supporters and opponents of the death penalty read invented studies with identical methods and opposite results, and both "rated the studies as better quality research when the results supported their views." Across the 51 studies the effect was modest in size (the authors call it small to medium), statistically reliable in 39 of them, and almost exactly as large among liberals as among conservatives.

Two cautions keep that in proportion. First, the symmetry is disputed. Jonathan Baron and John Jost replied that Ditto and colleagues were "too quick to embrace a false equivalence between the liberal left and the conservative right," and argued that other research suggests "epistemic virtues are more prevalent among liberals than conservatives." (We have read only their published summary.) Ditto and colleagues answered in the same issue, defending their choice of studies; the question remains open.

Second, a famous further claim has not held up in a large recent test. As Andrew Guess and Alexander Coppock describe the 1979 study, its subjects "reported becoming" more firmly attached to their original views after reading mixed evidence. Guess and Coppock re-ran the design with the original study texts, lightly updated, and measured each person's attitudes before and after reading. Across that and two other large experiments they found "no evidence of backlash, even under theoretically favorable conditions," and concluded that backlash is "much rarer than commonly supposed." People rate welcome evidence more kindly; unwelcome evidence does not, on average, make them more stubborn.

The exchange between the two teams also marks the tool's limit. Baron and Jost argued, in Ditto and colleagues' quotation of them, that it is "perfectly rational to evaluate new information on the basis of prior beliefs." Ditto and colleagues agreed that what you already believe should shape where you end up, but drew a line at "using consistency with one's priors to assess the validity of the new information." That line is the Mirror Check's. A surprising study may rightly leave you less convinced than a familiar one, because your other evidence still counts. What should not change is the questions you ask about how the study was done. The Mirror Check does not demand the same verdict on every claim. It demands the same questions.

Using it well

The legitimate use is applying to a claim that flatters your side the checking you would apply to one that did not, and letting the result stand.

Jonathan Capehart, a Washington Post opinion writer, did this in public. After the police shooting of Michael Brown in Ferguson, Missouri, in August 2014, he had, in his words, "decried the next morning the death of yet another unarmed black man at the hands of a white police officer." On March 16, 2015, after reading two Justice Department reports, he published a column headlined "'Hands up, don't shoot' was built on a lie." As The Advocate quoted it, he wrote that the storyline "isn't corroborated by ballistic and DNA evidence and multiple witness statements," and closed: "we must never allow ourselves to march under the banner of a false narrative on behalf of someone who would otherwise offend our sense of right and wrong. And when we discover that we have, we must acknowledge it, admit our error and keep on marching." A correction like this is the Mirror Check's last step made public: a claim he had welcomed was checked against the evidence, and his verdict moved.

His reading matches the summary in the Justice Department's report on the shooting of March 4, 2015. It concluded that the evidence "does not support charging a violation of federal law," and found that accounts of Brown holding his hands up in surrender were inconsistent with the physical and forensic evidence, inconsistent with the witnesses' own earlier statements, or later withdrawn. (We have read the report's introduction and summary, not its full witness-by-witness analysis, and not the second report Capehart cites. The report decided whether federal charges could be brought; this page takes no further position on the shooting.)

A case from the other side shows the check run halfway. On November 19, 2020, Tucker Carlson, then a host on the conservative Fox News channel, said in his evening monologue, as Fox News published it that night, that Sidney Powell, a lawyer who, as he described it, was claiming for President Trump that voting software had flipped millions of votes, had been asked for her proof: "she never sent us any evidence, despite a lot of polite requests." That is the Mirror Check applied to a claim that, by his own account, many Trump voters hoped was true. Yet the same piece said Rudy Giuliani "did not conclusively prove" fraud "but he did raise legitimate questions," and offered a rule of thumb that runs the other way: "The louder the Yale political science department and the staff of The Atlantic magazine scream 'conspiracy theory,' the more interested we tend to be." The demand for evidence was real. It was not yet even-handed. (One court's later finding on those claims is on record. In August 2021 a federal judge in Michigan sanctioned her and other lawyers for a lawsuit alleging that software had shifted votes from Trump to Biden, finding that they had brought "claims not backed by evidence." The judge said she was not deciding whether there had been fraud, only whether the lawyers had abused the court's process.)

When it becomes a weapon

The weaponized form is bulverism, a name coined by C. S. Lewis in an essay whose first version appeared in 1941: explaining away a belief by diagnosing the believer's motives instead of answering the claim. Lewis's rule, as quoted on an evangelical ministry's blog from the essay, is "You must show that a man is wrong before you start explaining why he is wrong." If you suspect a man's belief in his bank balance is wishful thinking, "Your only chance of finding out is to sit down and work through the sum yourself."

The trap is that bulverism sounds like the Mirror Check: both say that wanting a conclusion can bend reasoning. The Mirror Check turns the question of motive on yourself and sends you back to the evidence. Bulverism turns it on someone else and skips the evidence. A diagnosis of why someone believes something, however accurate, says nothing about whether it is true.

The move is found across politics, and it often arrives as a charge about money, which is also how the related tool D1: The Incentive Trace goes wrong: "funded by X" offered as if it meant "false." On August 17, 2011, Rick Perry, governor of Texas and a Republican candidate for president, explained his doubts about climate science at a breakfast in New Hampshire. CBS News reported him saying "there are a substantial number of scientists who have manipulated data so that they will have dollars rolling into their projects." He offered other reasons too, but the charge that carried his answer was that scientists had a motive to cheat. In the remarks CBS reported, he named no one and offered no evidence that anyone had.

Sanders's statement on the Mercatus study comes close to the same move from the left. Its first sentence argued from other countries' spending, but its only answer to the report's own figures was the charge about who had paid for it. His office had engaged the substance elsewhere: the Associated Press reported that his staff found an error in an early version, which the author corrected. Who pays for a study is a fair reason to check it harder. It is not a finding about its arithmetic.

Bulverism also tends to exempt the person doing it, and it has a subtler version: pointing out a real inconsistency ("you questioned the sample size on that study but not this one") is fair, but it becomes a weapon when "you have a double standard" is offered in place of an answer to whether the claim holds.

Some signs help tell the two apart:

• The legitimate form is run on your own reasoning first, usually in private. The weaponized form is aimed at an opponent, usually in public.
• The legitimate form ends with a return to the evidence. The weaponized form ends with a verdict on the person.
• The legitimate user expects to find the bias in themselves at least as often as in others. The weaponized user finds it only in others.

Run it on your own side first

Every tool on this site carries this instruction, and for this one it is most of the point. A Mirror Check run only on other people is not a Mirror Check. Try these:

• Think of the last claim you shared about a subject you care about. What did you check before you shared it? Would that have been enough if the claim had pointed the other way?
• In the Mercatus story above, did Sanders's thanks to the Koch brothers strike you as a fair point or a dodge? Would you have judged it the same way if a Republican had thanked a progressive group for a study that happened to help him?
• When did you last explain someone's view by their motives? Did you also answer the view?

If the honest answer to the first question is "I checked nothing," you are in large company: the experiments above found a version of this habit on the left and the right alike.

Grammars it works against

A grammar, on this site, is a recurring pattern of argument that turns up across many subjects. The Mirror Check is one of four tools this site treats as working against every grammar. Each grammar has two poles, and the Mirror Check asks whether you hold the pole you favor to the same standard as the one you oppose. It matters most in three.

• G8: Status Anxiety Dressed as Principle. Positions that protect someone's social standing are argued as pure principle. Run on yourself, the Mirror Check asks what your own principled stance protects for you. It also guards the other pole, the bulverism trap of explaining away every position as status-seeking.
• G4: Loyalty, Heresy, and Betrayal. Groups punish the dissenter inside harder than the opponent outside. The Mirror Check asks whether you are judging the heretic's argument or the fact that the heretic made it, and asks the same of the heretic who takes expulsion as proof of being right.
• G7: Moral Panic and the Folk Devil. Alarm about a group can run far ahead of the real harm, and the charge of "panic" can wave away a harm that is real. The Mirror Check asks whether you accept alarming evidence more readily about some groups than others, and reach for the word "panic" only when the alarm is about a group you sympathize with.

Related tools

• A5: The Situation Swap is the Mirror Check applied to explanations of behavior: we excuse our own side's conduct by circumstance and condemn the other side's identical conduct as character.
• B5: The Fallacy-Fallacy Guard shares the lesson about bulverism: catching a bad argument, or a bad motive, does not make the conclusion false.
• D1: The Incentive Trace asks who profits from your believing a claim. It is the fair cousin of bulverism: it adjusts how much checking a claim deserves, then still checks the claim.
• C2: The Steelman Protocol asks you to state the other side's position so well they would sign it. The Mirror Check makes sure you hold that strongest version to the same standard as your own.