AI System Reliability Tracking

Part of the Design Sketches for Collective Epistemics series by Forethought Foundation.

Overview

Reliability Tracking is a proposed system for systematically assessing the accuracy and trustworthiness of public actors—individuals, organizations, media outlets—by creating topic-specific track records rather than generalized reputation scores. The concept was outlined in Forethought Foundation's 2025 report "Design Sketches for Collective Epistemics."

The core problem: in current public discourse, bold predictions and confident factual claims face few consequences when they turn out to be wrong. A pundit who confidently predicts economic collapse every year suffers no reputational penalty; a company that repeatedly overpromises on product timelines faces no systematic accountability. Reliability tracking aims to "heal that feedback loop" by making track records visible and searchable.

Unlike a simple credibility score, the system would provide topic-specific assessments. Someone might be highly reliable on climate science but consistently wrong about economic predictions, or accurate about technical capabilities but unreliable about timelines.

How It Would Work

Step-by-Step Process

1. Compile database: Gather past public statements from articles, interviews, social media, reports, and press releases

2. Classify and timestamp: Identify specific claims—factual assertions, forward-looking predictions, concrete promises—and record them with dates

3. Score trustworthiness via LLM evaluation:

   • Do factual claims match primary sources?
   • Did predictions match subsequent events?
   • Were promises kept in a timely manner?

4. Aggregate scores by claim type and topic area, with user-customizable methodology

5. Detect patterns: Identify where an actor is consistently reliable or unreliable (e.g., "reliable on topic X, consistently overpromises on Y")

Three Types of Claims Tracked

Browser Integration

The envisioned user experience includes:

• A browser widget that provides reliability ratings when viewing content from tracked actors
• Health warnings displayed for sources with poor track records in the relevant topic area
• Drill-down capability to browse specific evaluations, see the methodology, and adjust scoring parameters
• User-customizable methodology: Different users might weight different factors, choose different trusted sources for ground truth, or set different thresholds

Design Challenges

Forethought identifies several significant design challenges:

Vague Claims

Many public statements are deliberately vague, making evaluation difficult:

Ground Truth Determination

Not all facts are uncontested:

Gameability

Reliability tracking systems are inherently gameable:

Cost Considerations

Forethought estimates that assessing a single person's track record costs "between cents and hundreds of dollars," depending on:

Existing Work and Precursors

Prediction Tracking Platforms

Several existing platforms implement aspects of reliability tracking:

Pundit and Public Figure Accountability Projects

Individual Track Record Practices

Some individuals have established personal accountability norms that demonstrate the concept:

• Scott Alexander: Since 2014, publishes annual predictions with explicit probabilities on Astral Codex Ten, then publicly scores them at year's end with calibration results
• Coefficient Giving: Published "How Accurate Are Our Predictions?" evaluating their own internal prediction calibration

Research on Accountability and Calibration

Key research findings that support reliability tracking:

• Tetlock (2005): 20-year study with 284 experts making about 28,000 predictions. Average expert "roughly as accurate as a dart-throwing chimpanzee." "Foxes" (drawing on many frameworks) consistently outperformed "hedgehogs." Published as Expert Political Judgment (Princeton University Press).
• IARPA ACE Tournament (2011-2015): Tetlock's Good Judgment Project beat all competing teams by 35-72%. Top 2% "superforecasters" were 30% more accurate than professional intelligence analysts with classified information access.
• Mellers et al. (2014): Identified superforecasters who are consistently well-calibrated across domains, suggesting reliability is a somewhat stable trait
• Atanasov et al. (2016): Tracking and feedback on predictions improves individual calibration by 20-30%
• DellaVigna & Pope (2018): Expert predictions about behavioral interventions were poorly calibrated, suggesting need for systematic tracking
• Brier Score: The standard proper scoring rule for probabilistic predictions (0=perfect, 1=worst). Incentivizes honest forecasting by simultaneously measuring calibration and resolution.

Calibration Training Tools

Several tools exist specifically to improve individual calibration:

Niche Applications

Forethought suggests several high-value starting points:

AI Lab Claim Tracking

A particularly relevant application for AI safety would be tracking the reliability of AI lab statements about:

• Capability timelines ("We'll achieve X by date Y")
• Safety commitments ("We won't release models above threshold Z")
• Benchmark claims ("Our model achieves state-of-the-art on benchmark B")
• Risk assessments ("This model poses minimal risk of X")

This could provide empirical grounding for debates about AI governance by making it visible which organizations consistently overpromise, underdeliver on safety commitments, or make accurate predictions about capabilities.

Worked Example: Tracking AI Lab Timeline Predictions

Consider tracking the reliability of a prominent AI lab CEO who has made repeated public predictions about AI capabilities:

Statement database (sampled):

Topic-specific reliability scores:

Browser widget display: When this CEO's next blog post appears, users would see: "Reliability: Mixed. This source is generally accurate on technical details but consistently overpromises on timelines (average 2 years early) and has made multiple safety claims later contradicted by evidence. [See full assessment →]"

This example illustrates both the power (concrete accountability) and the difficulty (judgment calls on "partially correct," handling predictions not yet resolved, legal risks of publishing low scores for named individuals).

Extensions and Open Ideas

Confidence-weighted prediction tracking: Rather than just right/wrong, score predictions based on the confidence expressed. Someone who says "I'm 90% confident AGI arrives by 2027" and is wrong should lose more credibility than someone who said "there's maybe a 30% chance." This connects to forecasting best practices and rewards epistemic humility.

Prediction network analysis: Map who influences whose predictions. If 10 commentators all predicted the same thing and were all wrong, did they independently reach the same conclusion, or did one person's prediction cascade through the network? Tracking influence chains helps identify independent vs. correlated errors.

"Reliability API": Provide a public API that other tools can query. A browser could check the reliability score of every author on a page. An AI chatbot could weight sources by their reliability scores when doing retrieval. Community Notes could use reliability data to prioritize which content to annotate.

Self-tracking mode: Let individuals track their own reliability privately before any public accountability. People often overestimate their prediction accuracy. A tool that says "You thought you were right 85% of the time, but you were actually right 62%" could improve personal calibration without the social pressure of public scores.

Temporal decay weighting: Old predictions should matter less than recent ones. Someone who made bad predictions 10 years ago but has been well-calibrated for the last 3 years should have a higher current score. This incentivizes improvement rather than creating permanent reputation damage.

Institutional reliability tracking: Track organizations (labs, think tanks, government agencies, media outlets) in addition to individuals. This sidesteps some privacy concerns while still providing useful information. "This think tank's economic predictions have been right 40% of the time over the last decade."

"Steelman" mode: When displaying a poor reliability score, also show the subject's best predictions and most honest moments. This reduces the adversarial feel and encourages engagement rather than defensive reactions.

Cross-reference with prediction markets: Automatically compare a person's stated predictions with the contemporaneous prediction market price. "This person predicted X with 90% confidence, but the prediction market at the time was at 35%. Their prediction was both confident and contrarian—and turned out to be correct." This highlights when people add genuine information vs. repeating conventional wisdom.

Challenges and Risks

Social Challenges

• Trust in the tracker: Unreliable sources may attack the tracking system rather than improve their accuracy
• Legal exposure: Public reliability scores could face defamation lawsuits, especially from well-resourced actors
• Adoption: Requires sufficient users to create social pressure for accuracy
• Polarization: Risk that reliability tracking becomes a political weapon

Technical Challenges

• Statement attribution: Correctly attributing claims to the right person/organization
• Interpretation: Same statement can be interpreted differently
• Scale: Tracking all public statements from all actors is computationally expensive
• Temporal dynamics: Beliefs should update; how do you handle changing views?

Ethical Challenges

• Privacy: Tracking statements of private individuals raises consent issues
• Power asymmetry: More public = more tracked; could disproportionately scrutinize certain groups
• Context collapse: Statements made in specific contexts may be unfairly evaluated out of context
• Rehabilitation: How do people escape bad track records? Is there a path to redemption?

Connection to AI Safety

Reliability tracking is directly relevant to several aspects of the AI transition model:

• Epistemic healthAi Transition Model ParameterEpistemic HealthThis page contains only a component placeholder with no actual content. Cannot be evaluated for AI prioritization relevance.: Making track records visible creates accountability for claims about AI capabilities, risks, and timelines
• Civilizational competenceAi Transition Model FactorCivilizational CompetenceSociety's aggregate capacity to navigate AI transition well—including governance effectiveness, epistemic health, coordination capacity, and adaptive resilience.: Better-calibrated public discourse about AI improves the quality of governance decisions
• Societal trustAi Transition Model ParameterSocietal TrustThis page contains only a React component placeholder with no actual content rendered. No information about societal trust as a factor in AI transition is present.: Trust based on track records is more robust than trust based on authority or charisma
• Lab accountability: Systematic tracking of AI lab claims could be a lightweight governance mechanism that complements formal regulation

Key Uncertainties

Further Reading

• Original Report: Design Sketches for Collective Epistemics — Reliability Tracking — Forethought Foundation
• Key Research: Expert Political Judgment by Philip Tetlock (2005) — foundational work on prediction accuracy and calibration
• Prediction Platforms: MetaculusOrganizationMetaculusMetaculus is a reputation-based forecasting platform with 1M+ predictions showing AGI probability at 25% by 2027 and 50% by 2031 (down from 50 years away in 2020). Analysis finds good short-term ca...Quality: 50/100, Prediction MarketsApproachPrediction Markets (AI Forecasting)Prediction markets achieve Brier scores of 0.16-0.24 (15-25% better than polls) by aggregating dispersed information through financial incentives, with platforms handling $1-3B annually. For AI saf...Quality: 56/100
• Overview: Design Sketches for Collective Epistemics — parent page with all five proposed tools