Back
OpenAI CoT Monitoring
webCredibility Rating
4/5
High(4)High quality. Established institution or organization with editorial oversight and accountability.
Rating inherited from publication venue: OpenAI
An OpenAI safety research page on using chain-of-thought reasoning traces as a monitoring and alignment verification tool, relevant to scalable oversight and detecting deceptive or misaligned model behavior in deployed systems.
Metadata
Importance: 72/100blog postprimary source
Summary
OpenAI describes their approach to monitoring chain-of-thought (CoT) reasoning in large language models as a safety measure, examining whether models' visible reasoning steps are faithful and can detect deceptive or misaligned behavior. The work explores using CoT transparency as a tool for alignment verification and identifying specification gaming or reward hacking patterns before deployment.
Key Points
- •Chain-of-thought reasoning can serve as a monitoring signal to detect whether models are pursuing unintended or deceptive strategies
- •Visible reasoning steps may reveal misalignment between stated objectives and actual model behavior, enabling early intervention
- •CoT monitoring is proposed as part of a broader scalable oversight strategy to maintain human visibility into model decision-making
- •The approach investigates whether models' internal reasoning is faithful enough to be trusted as a safety signal
- •Relevant to detecting reward hacking and specification gaming by catching problematic reasoning before it manifests in actions
Cited by 3 pages
| Page | Type | Quality |
|---|---|---|
| Reward Hacking Taxonomy and Severity Model | Analysis | 71.0 |
| Eval Saturation & The Evals Gap | Approach | 65.0 |
| Scalable Eval Approaches | Approach | 65.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 202615 KB
OpenAI
March 10, 2025
[Publication](https://openai.com/research/index/publication/)
# Detecting misbehavior in frontier reasoning models
Frontier reasoning models exploit loopholes when given the chance. We show we can detect exploits using an LLM to monitor their chains-of-thought. Penalizing their “bad thoughts” doesn’t stop the majority of misbehavior—it makes them hide their intent.
[Read paper(opens in a new window)](https://arxiv.org/abs/2503.11926)
Loading…
Share
Chain-of-thought (CoT) reasoning models “think” in natural language understandable by humans. Monitoring their “thinking” has allowed us to detect misbehavior such as subverting tests in coding tasks, deceiving users, or giving up when a problem is too hard.
We believe that CoT monitoring may be one of few tools we will have to oversee superhuman models of the future.
We have further found that directly optimizing the CoT to adhere to specific criteria (e.g. to not think about reward hacking) may boost performance in the short run; however, it does not eliminate all misbehavior and can cause a model to hide its intent. We hope future research will find ways to directly optimize CoTs without this drawback, but until then
We recommend against applying strong optimization pressure directly to the CoTs of frontier reasoning models, leaving CoTs unrestricted for monitoring.
We understand that leaving CoTs unrestricted may make them unfit to be shown to end-users, as they might violate some misuse policies. Still, if one wanted to show policy-compliant CoTs directly to users while avoiding putting strong supervision on them, one could use a separate model, such as a CoT summarizer or sanitizer, to accomplish that.
Humans often find and exploit loopholes—whether it be sharing online subscription accounts against terms of service, claiming subsidies meant for others, interpreting regulations in unforeseen ways, or even lying about a birthday at a restaurant to get free cake. Designing robust reward structures that do not inadvertently incentivize unwanted behavior is remarkably hard, and it isn’t a problem limited to human institutions; it’s also one of the core challenges in developing capable, aligned AI systems. In reinforcement learning settings, exploiting unintended loopholes is commonly known as **reward hacking**, a phenomenon where AI agents achieve high rewards through behaviors that don’t align with the intentions of their designers.
Since even humans (who by definition have human-level intelligence) reward hack systems, simply continuing to push the frontier of AI model intelligence likely won’t solve the issue. In fact, enhancing AI agent capabilities may exacerbate the problem by better equipping them to discover and execute more complex and hard-to-monitor exploits. As we’ve
... (truncated, 15 KB total)Resource ID:
d4700c15258393ad | Stable ID: NTk4M2Q2Nz