TruthfulQA: Benchmark for Measuring Truthfulness in Language Models

web

Credibility Rating

3/5

Good(3)

Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.

Rating inherited from publication venue: GitHub

TruthfulQA is a widely cited benchmark in AI safety for evaluating model honesty; it is frequently used to assess RLHF and fine-tuning approaches aimed at reducing hallucination and sycophancy in LLMs.

Metadata

Importance: 78/100dataset

Summary

TruthfulQA is a benchmark dataset designed to measure whether language models generate truthful answers to questions. It contains 817 questions across 38 categories where humans often hold false beliefs, testing whether LLMs reproduce common misconceptions. The benchmark highlights that larger models are not necessarily more truthful and can be confidently wrong.

Key Points

•Contains 817 questions designed to elicit false answers that humans commonly believe, covering topics like health, law, finance, and conspiracy theories
•Finds that larger language models tend to perform worse on truthfulness, suggesting scale alone does not solve honesty problems
•Introduces two evaluation metrics: truthfulness (factual accuracy) and informativeness (avoiding evasive non-answers)
•Best-performing models at release achieved ~58% truthfulness vs 94% for humans, revealing a significant gap
•Serves as a foundational evaluation tool for AI alignment work on honesty, deception, and calibration

Cited by 1 page

Page	Type	Quality
Sycophancy	Risk	65.0

Cached Content Preview

HTTP 200Fetched Mar 20, 202618 KB

[Skip to content](https://github.com/sylinrl/TruthfulQA#start-of-content)

You signed in with another tab or window. [Reload](https://github.com/sylinrl/TruthfulQA) to refresh your session.You signed out in another tab or window. [Reload](https://github.com/sylinrl/TruthfulQA) to refresh your session.You switched accounts on another tab or window. [Reload](https://github.com/sylinrl/TruthfulQA) to refresh your session.Dismiss alert

{{ message }}

[sylinrl](https://github.com/sylinrl)/ **[TruthfulQA](https://github.com/sylinrl/TruthfulQA)** Public

- [Notifications](https://github.com/login?return_to=%2Fsylinrl%2FTruthfulQA) You must be signed in to change notification settings
- [Fork\\
112](https://github.com/login?return_to=%2Fsylinrl%2FTruthfulQA)
- [Star\\
894](https://github.com/login?return_to=%2Fsylinrl%2FTruthfulQA)


main

[**2** Branches](https://github.com/sylinrl/TruthfulQA/branches) [**0** Tags](https://github.com/sylinrl/TruthfulQA/tags)

[Go to Branches page](https://github.com/sylinrl/TruthfulQA/branches)[Go to Tags page](https://github.com/sylinrl/TruthfulQA/tags)

Go to file

Code

Open more actions menu

## Folders and files

| Name | Name | Last commit message | Last commit date |
| --- | --- | --- | --- |
| ## Latest commit<br>[![sylinrl](https://avatars.githubusercontent.com/u/59480158?v=4&size=40)](https://github.com/sylinrl)[sylinrl](https://github.com/sylinrl/TruthfulQA/commits?author=sylinrl)<br>[Update README.md](https://github.com/sylinrl/TruthfulQA/commit/d71c110897f5d31c5d7f309e7bc316c152f6f031)<br>last yearJan 15, 2025<br>[d71c110](https://github.com/sylinrl/TruthfulQA/commit/d71c110897f5d31c5d7f309e7bc316c152f6f031) · last yearJan 15, 2025<br>## History<br>[36 Commits](https://github.com/sylinrl/TruthfulQA/commits/main/) <br>Open commit details<br>[View commit history for this file.](https://github.com/sylinrl/TruthfulQA/commits/main/) 36 Commits |
| [data](https://github.com/sylinrl/TruthfulQA/tree/main/data "data") | [data](https://github.com/sylinrl/TruthfulQA/tree/main/data "data") | [adding binary MC option](https://github.com/sylinrl/TruthfulQA/commit/f6be04e52bbcb41d4d20daee6358231d4a5015d2 "adding binary MC option") | last yearJan 14, 2025 |
| [truthfulqa](https://github.com/sylinrl/TruthfulQA/tree/main/truthfulqa "truthfulqa") | [truthfulqa](https://github.com/sylinrl/TruthfulQA/tree/main/truthfulqa "truthfulqa") | [String formatting fix](https://github.com/sylinrl/TruthfulQA/commit/27a13fad4ca91a3cf4ac82009615067b95723fca "String formatting fix") | 5 years agoSep 17, 2021 |
| [.gitignore](https://github.com/sylinrl/TruthfulQA/blob/main/.gitignore ".gitignore") | [.gitignore](https://github.com/sylinrl/TruthfulQA/blob/main/.gitignore ".gitignore") | [tidy](https://github.com/sylinrl/TruthfulQA/commit/25fca9e323f22c7bc02b2f9d910d507ba04d045d "tidy") | 5 years agoAug 25, 2021 |
| [LICENSE](https://github.com/sylinrl/TruthfulQA/blob/main/LICENSE "LICENSE") | [LICENSE](https://github.com/sylinrl/TruthfulQA/bl

... (truncated, 18 KB total)

Resource ID: f37142feae7fe9b1 | Stable ID: OTkwZWFkM2