Back
International Fact-Checking Network
webmisinforeview.hks.harvard.edu·misinforeview.hks.harvard.edu/article/fact-checking-fact-...
Relevant to AI safety insofar as fact-checking networks inform training data quality and AI-assisted content moderation; understanding their reliability and biases matters for downstream AI systems that rely on them.
Metadata
Importance: 38/100organizational reportanalysis
Summary
This Harvard Misinformation Review article presents a data-driven evaluation of fact-checking organizations, assessing their methodologies, consistency, and reliability. It examines whether fact-checkers themselves meet standards of accuracy and transparency, providing an empirical framework for auditing information quality gatekeepers. The work is relevant to understanding the infrastructure of epistemic quality control in public discourse.
Key Points
- •Applies quantitative, data-driven methods to evaluate the quality and consistency of fact-checking organizations themselves.
- •Examines whether fact-checkers adhere to stated methodological standards, surfacing potential biases or inconsistencies.
- •Provides a meta-level audit framework useful for assessing the reliability of information quality institutions.
- •Relevant to debates about who watches the watchmen in information ecosystems and epistemic infrastructure.
- •Findings have implications for how AI systems trained on fact-checked data inherit or propagate systematic biases.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| AI-Era Epistemic Infrastructure | Approach | 59.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 202642 KB
[Skip to main content](https://misinforeview.hks.harvard.edu/article/fact-checking-fact-checkers-a-data-driven-approach/#main-content)
_This study examined four fact checkers (Snopes, PolitiFact, Logically, and the Australian Associated Press FactCheck) using a data-driven approach. First, we scraped 22,349 fact-checking articles from Snopes and PolitiFact and compared their results and agreement on verdicts. Generally, the two fact checkers agreed with each other, with only one conflicting verdict among 749 matching claims after adjusting minor rating differences. Next, we assessed 1,820 fact-checking articles from Logically and the Australian Associated Press FactCheck and highlighted the differences in their fact-checking behaviors. Major events like the COVID-19 pandemic and the presidential election drove increased the frequency of fact-checking, with notable variations in ratings and authors across fact checkers._
###### By
[Sian Lee](https://misinforeview.hks.harvard.edu/article/author/sian-lee "Posts by Sian Lee")
College of Information Sciences and Technology, The Pennsylvania State University, USA
[Aiping Xiong](https://misinforeview.hks.harvard.edu/article/author/aiping-xiong "Posts by Aiping Xiong")
College of Information Sciences and Technology, The Pennsylvania State University, USA
[Haeseung Seo](https://misinforeview.hks.harvard.edu/article/author/haeseung-seo "Posts by Haeseung Seo")
College of Information Sciences and Technology, The Pennsylvania State University, USA
[Dongwon Lee](https://misinforeview.hks.harvard.edu/article/author/dongwon-lee "Posts by Dongwon Lee")
College of Information Sciences and Technology, The Pennsylvania State University, USA
###### Article Metrics
17
CrossRef Citations
1405
PDF Downloads
132653
Page Views
Image by [Geralt](https://pixabay.com/users/geralt-9301/ "") on [Pixabay](https://pixabay.com/ "")
###### Topics
- [Debunking](https://misinforeview.hks.harvard.edu/explore/?fwp_topic=debunking "Debunking")
- / [Fact-checking](https://misinforeview.hks.harvard.edu/explore/?fwp_topic=fact-checking "Fact-checking")
- / [Platforms](https://misinforeview.hks.harvard.edu/explore/?fwp_topic=platforms "Platforms")
## Research Questions
- Do different fact checkers exhibit similar or distinct behaviors with respect to the frequency of fact-checking, types of claims selected for fact-checking, and the individuals responsible for conducting fact checks?
- What percentage of statements debunked by fact checkers are overlapping (i.e., matching claims) across multiple fact checkers?
- Is there a reasonable level of agreement among fact checkers in their ratings of matching claims that have been debunked by multiple fact checkers?
## Essay Summary
- This study examined four fact-checking orga
... (truncated, 42 KB total)Resource ID:
a9e552996d830d81 | Stable ID: MzMzNjAxNm