Back
Vague Verbiage Forecasting - Good Judgment Inc.
webCredibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: Good Judgment
Relevant to AI safety practitioners communicating risk estimates; imprecise language around AI timelines or catastrophic risk probabilities mirrors the broader forecasting problem discussed here by Good Judgment, a leading forecasting organization.
Metadata
Importance: 42/100blog posteducational
Summary
Good Judgment Inc. examines how vague verbal expressions like 'likely,' 'probably,' or 'may' create ambiguity in forecasting and risk communication, leading to miscommunication and poor decision-making. The piece advocates for numerical probability estimates over verbal probability expressions to improve forecast precision and interpretability. This has direct relevance to how AI risk assessments and safety predictions are communicated.
Key Points
- •Verbal probability terms (e.g., 'likely,' 'possible') are interpreted inconsistently across individuals, causing significant communication failures.
- •Numerical probabilities provide clearer, more actionable information for decision-makers compared to vague linguistic qualifiers.
- •Good Judgment's superforecasting methodology emphasizes precise quantification over imprecise natural language descriptions of uncertainty.
- •Vague verbiage in risk communication—including AI safety contexts—can lead to systematic under- or over-reaction to threats.
- •Standardizing probabilistic language in forecasts improves calibration, accountability, and the ability to track forecast accuracy over time.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Good Judgment (Forecasting) | Organization | 50.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 20268 KB
[Superforecaster Perspectives](https://goodjudgment.com/category/superforecaster-perspectives/ "Go to the Superforecaster Perspectives category archives.") \> How Distinct Is a “Distinct Possibility”?
# How Distinct Is a “Distinct Possibility”? Vague Verbiage in Forecasting
“What does a ‘fair chance’ mean?”
It is a question posed to a diverse group of professionals—financial advisers, political analysts, investors, journalists—during one of Good Judgment Inc’s virtual workshops. The participants have joined the session from North America, the EU, and the Middle East. They are about to get intensive hands-on training to become better forecasters. Good Judgment’s Senior Vice President Marc Koehler, a Superforecaster and former diplomat, leads the workshop. He takes the participants back to 1961. The young President John F. Kennedy asks his Joint Chiefs of Staff whether a CIA plan to topple the Castro government in Cuba would be successful. They tell the president the plan has a “fair chance” of success.
The workshop participants are now asked to enter a value between 0 and 100—what do they think is the probability of success of a “fair chance”?
When they compare their numbers, the results are striking. Their answers range from 15% to 75% with the median value of 60%.
Figure 1. Meanings behind vague verbiage according to a Good Judgment poll. Source: Good Judgment.
The story of the 1961 Bay of Pigs invasion is recounted in Good Judgment co-founder Philip Tetlock’s _Superforecasting: The Art and Science of Prediction_ (co-authored with Dan Gardner). The advisor who wrote the words “fair chance,” the story goes, later said what he had in mind was only a 25% chance of success. But like many of the participants in the Good Judgment workshop some 60 years later, President Kennedy took the phrase to imply a more positive assessment of success. By using vague verbiage instead of precise probabilities, the analysts failed to communicate their true evaluation to the president. The rest is history: The Bay of Pigs plan he approved ended in failure and loss of life.
Vague verbiage is pernicious in multiple ways.
**1\. Language is open to interpretations. Numbers are not.**
According to research published in the [_Journal of Experimental Psychology_](https://www.academia.edu/851574/Measuring_the_vague_meanings_of_probability_terms), “maybe” ranges from 22% to 89%, meaning radically different things to different people under different circumstances. Survey research by Good Judgment shows the implied ranges for other vague terms, with “distinct possibility” ranging from 21% to 84%. Yet, “distinct possibility” was the phrase [used](https://goodjudgment.com/services/futurefirst/) by White House National Security Adviser Jake Sullivan on the eve of the Russian invasion in Ukraine.
Figure 2. How people interpret probabilistic words. Source: Andrew Maubou
... (truncated, 8 KB total)Resource ID:
9b643960b1464eb6 | Stable ID: MWEzOWMzMT