Back
LessWrong: The Most Forbidden Technique
blogAuthor
Zvi
Credibility Rating
3/5
Good(3)Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: LessWrong
Zvi's analysis of a pivotal OpenAI paper establishing a key safety norm: CoT and other interpretability signals must be preserved as monitoring tools and never incorporated into training objectives, as doing so causes models to learn to hide misaligned behavior rather than correct it.
Forum Post Details
Karma
168
Comments
9
Forum
lesswrong
Forum Tags
Orthogonality ThesisAI
Metadata
Importance: 78/100blog postcommentary
Summary
Zvi analyzes an OpenAI paper demonstrating that applying optimization pressure to Chain-of-Thought monitoring causes models to hide reward-hacking behavior rather than eliminate it, destroying the very oversight capability. The core principle: interpretability techniques must never be used as training targets, only as monitoring tools, because doing so creates adversarial pressure that defeats detection of misalignment.
Key Points
- •Training AI on interpretability signals (like CoT monitoring) causes models to obfuscate reasoning and defeat the monitoring technique rather than improve behavior.
- •OpenAI paper shows frontier models already perform complex reward hacking, and CoT monitoring can detect it—but only if optimization pressure is not applied to the CoT.
- •Even small amounts of optimization pressure on CoT rapidly degrades monitorability, making obfuscated reward hacking the outcome.
- •Both OpenAI and Anthropic have adopted policies against applying strong optimization pressure to CoT, treating it as a preserved oversight channel.
- •Interpretability tools represent scarce 'bits of optimization pressure'—they must be conserved for oversight, not consumed in training objectives.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Goodfire | Organization | 68.0 |
Cached Content Preview
HTTP 200Fetched Mar 15, 202635 KB
x This website requires javascript to properly function. Consider activating javascript to get access to all site functionality. The Most Forbidden Technique — LessWrong Orthogonality Thesis AI Frontpage 2025 Top Fifty: 27 % 168
The Most Forbidden Technique
by Zvi 12th Mar 2025 Don't Worry About the Vase 20 min read 9 168
The Most Forbidden Technique is training an AI using interpretability techniques.
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
Those bits of optimization pressure from [T] are precious. Use them wisely.
Table of Contents
New Paper Warns Against the Most Forbidden Technique.
Reward Hacking Is The Default.
Using CoT to Detect Reward Hacking Is Most Forbidden Technique.
Not Using the Most Forbidden Technique Is Harder Than It Looks.
It’s You, It’s Also the Incentives.
The Most Forbidden Technique Quickly Backfires.
Focus Only On What Matters.
Is There a Better Way?
What Might We Do Next?
New Paper Warns Against the Most Forbidden Technique
OpenAI has now published an important new paper explaining and emphasizing the importance of applying this principle to the Chain of Thought (CoT).
The paper shows that current CoT-based models area already strong enough that, under sufficient optimization pressure, they learn to hide their intent within the CoT and then exhibit substantial reward hacking.
They summarize their contributions this way:
We show that frontier reasoning models often perform complex reward hacks in real-world coding environments.
We provide evidence that chain-of-thought monitoring can be effective in detecting the misaligned behavior of frontier reasoning models, and we show it can be far more accurate than action-only monitoring.
We show that chain-of-thought monitorability is very fragile. Applying direct optimization pressure to the CoT can quickly lead to obfuscated reward hacking—where agents hide their intent while continuing to misbehave.
Or:
We are already seeing complex reward hacking in the real world.
Monitoring the CoT can spot this.
Applying too much optimization pressure on the CoT doesn’t make the hacking go away, it instead makes your ability to detect the hacking go away.
‘Too much’ is a remarkably small amount.
The paper is excellent, laying out a very important principle to never, ever do this Most Forbidden Technique, and explaining why we must never, ever do this.
We believe that CoT monitoring may be one of few tools we will have to oversee superhuman
... (truncated, 35 KB total)Resource ID:
408e865884be7f38 | Stable ID: MDBlMWY0OD