Fact checking the fact checkers


fact checkersThere are various well-established and well-known fact checkers that many rely upon, for example PolitiFact, FactCheck.org, or the Washington Post’s Fact Checker. However, one has to wonder if anybody independently fact-checks the fact checkers.

I was curious about this and so looked into it to see if anybody had seriously researched it. To be honest, I would have been astonished to discover that nobody had. That however is not how things panned out because indeed yes it is a topic of interest.

The 2015 Study

Published within the Journal of Politics and Society is a paper entitled “Revisiting the Epistemology of Fact-Checking”. This is a response to a previous 2013 paper by Joseph E. Uscinski and Ryden W. Butler which had argued that fact-checking was useless because the methods fact-checkers used to select statements, consider evidence, and render judgment fails to stand up to the rigors of scientific inquiry.

This 2015 paper successfully argues that the previous paper was seriously flawed because they had sampled from sources that do not consistently fact-check on a regular basis and also that many of the criticisms it raised were not actually backed up by any empirical evidence.

What exactly did the study do?

The looked at political ads for the 2008 and 2012 presidential elections …

political ads from the 2008 and 2012 presidential election serve as the framework within which fact-checking is examined. Because they offer a bounded source of planned, strategically crafted messaging by political figures, political ads minimize the likelihood of off-the-cuff remarks or gaffes. Other forms of political communication can be less predictable with interruptions or opportunities to misspeak as well as be less precise

They then evaluated three very specific fact-checking organisations and eliminated others because they had no consistent methodology or were no longer fact-checking …

Ad claim accuracy is based upon the evaluations of FactCheck.org, PolitiFact.com, and the Washington Post’s Fact Checker. Other organizations such as Spinsanity and newspapers such as the New York Times have conducted fact-checking. However, Spinsanity disbanded prior to the 2008 election, and the political ad coverage of the Times did not use consistent methods to assess the accuracy of the ads

they are enduring fact-checking organizations on a national level that continue to operate beyond specific election cycles. Furthermore, they are considered the three elite, national fact-checkers

OK, so what exactly did they discover?

Well,  this is what they examined …

Among the 491 collected television ads from both elections, the three fact-checkers published evaluations of 192 of them. Many of the ads, however, were only evaluated by a single fact-checker. Across both elections, only 65 ads drew evaluations from at least two of the fact- checkers. Within these 65 ads, 150 different claims were scrutinized.

… and the result of this analysis is that they all more or less agreed. To be precise, the percentage of agreement was in the range 98-100%

That is very encouraging.

They do not conspire to align and instead independently reached more or less the exact same conclusions using distinctly different ways of fact checking.

What this does not tell us is that all fact-checkers get it right all the time, but instead, when it comes to evaluating claims in political ads from presidential races, the leading U.S. fact-checkers are highly consistent in their assessments.

Who are the popular Fact-checkers today?

In a US context, while there are of course others, it is basically these that are the mainstream primary go-to sources …

FactCheck.org

This is non-partisan, and nonprofit website that is self-described as “advocates for voters that aims to reduce the level of deception and confusion in U.S. politics,

Fact Checker (The Washington Post)

A project of The Washington Post, known for grading politicians on the factual accuracy of their statements with one to four “Pinocchios.”

  • Created September 2007 by Post diplomatic writer Michael Dobbs specifically for the 2008 presidential campaign.
  • Ceased operation 4 November 2008.
  • Relaunched with a broader focus in January 2011, led by veteran Post diplomatic correspondent Glenn Kessler.
  • Rates statements by politicians, usually on a range of one to four Pinocchios—with one Pinocchio for minor shading of the facts and four Pinocchios for outright lies. If the statement is truthful, the person will get a rare “Geppetto.”

PolitiFact.com

A service of the Tampa Bay Times

  • Created August 2007, it uses the “Truth-o-Meter” to rank the amount of truth in public persons’ statements.
  • 2009 Pulitzer Prize Winner.

Snopes.com

Focuses on, but is not limited to, validating and debunking urban legends and other stories in American popular culture.

TruthOrFiction.com

Validates and debunks urban legends, Internet rumors, e-mail forwards, and other stories of unknown or questionable origin.

Further Reading

Last year I wrote a posting in which I picked my personal selection of the top 5 fact checking websites. Not quite the above five, and also it explained not only what they did, but why I personally found them useful.

Leave a Reply