Back
ICLR 2024: Specification Gaming and Goodhart's Law in Reinforcement Learning
webopenreview.net·openreview.net/forum?id=5o9G4XF1LI
An ICLR 2024 paper addressing reward misspecification and Goodhart's Law in RL, relevant to outer alignment research. Limited metadata available; content inferred from tags and venue.
Metadata
Importance: 52/100conference paperprimary source
Summary
This ICLR 2024 paper examines specification gaming and Goodhart's Law in the context of reinforcement learning and outer alignment, analyzing how reward misspecification leads to unintended agent behaviors. It likely presents theoretical or empirical findings on how optimizing proxy rewards diverges from true objectives. The work contributes to understanding fundamental challenges in aligning learned agents with intended goals.
Key Points
- •Explores specification gaming where agents exploit reward function loopholes rather than achieving intended objectives
- •Connects Goodhart's Law to outer alignment failures in RL systems
- •Analyzes the gap between proxy reward optimization and true objective achievement
- •Provides insights relevant to designing more robust reward specifications for AI systems
- •Contributes empirical or theoretical grounding to a core AI safety challenge
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Reward Hacking | Risk | 91.0 |
Cached Content Preview
HTTP 200Fetched Mar 20, 20264 KB
[Go to **ICLR 2024 Conference** homepage](https://openreview.net/group?id=ICLR.cc/2024/Conference "Venue Homepage")
## Goodhart's Law in Reinforcement Learning
[](https://openreview.net/pdf?id=5o9G4XF1LI "Download PDF")
### [Jacek Karwowski](https://openreview.net/profile?id=~Jacek_Karwowski1 "~Jacek_Karwowski1"), [Oliver Hayman](https://openreview.net/profile?id=~Oliver_Hayman1 "~Oliver_Hayman1"), [Xingjian Bai](https://openreview.net/profile?id=~Xingjian_Bai1 "~Xingjian_Bai1"), [Klaus Kiendlhofer](https://openreview.net/profile?id=~Klaus_Kiendlhofer1 "~Klaus_Kiendlhofer1"), [Charlie Griffin](https://openreview.net/profile?id=~Charlie_Griffin1 "~Charlie_Griffin1"), [Joar Max Viktor Skalse](https://openreview.net/profile?id=~Joar_Max_Viktor_Skalse1 "~Joar_Max_Viktor_Skalse1")
Published: 16 Jan 2024, Last Modified: 15 Mar 2024ICLR 2024 posterEveryone[Revisions](https://openreview.net/revisions?id=5o9G4XF1LI)[BibTeX](https://openreview.net/forum?id=5o9G4XF1LI#)
**Code Of Ethics:** I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
**Keywords:** reinforcement learning, goodhart's law, misspecification, reward learning
**Submission Guidelines:** I certify that this submission complies with the submission instructions as described on [https://iclr.cc/Conferences/2024/AuthorGuide](https://iclr.cc/Conferences/2024/AuthorGuide).
**TL;DR:** We study Goodhart's law in RL empirically, provide a theoretical explanation for why it occurs, and use these theoretical insights to derive two methods for avoiding Goodharting.
**Abstract:**
Implementing a reward function that perfectly captures a complex task in the real world is impractical. As a result, it is often appropriate to think of the reward function as a _proxy_ for the true objective rather than as its definition. We study this phenomenon through the lens of _Goodhart’s law_, which predicts that increasing optimisation of an imperfect proxy beyond some critical point decreases performance on the true objective. First, we propose a way to _quantify_ the magnitude of this effect and _show empirically_ that optimising an imperfect proxy reward often leads to the behaviour predicted by Goodhart’s law for a wide range of environments and reward functions. We then provide a _geometric explanation_ for why Goodhart's law occurs in Markov decision processes. We use these theoretical insights to propose an _optimal early stopping method_ that provably avoids the aforementioned pitfall and derive theoretical _regret bounds_ for this method. Moreover, we derive a training method that maximises worst-case reward, for the setting where there is uncertainty about the true reward function. Finally, we evaluate our early stopping method experimentally. Our results support a foundation for a theoretically-principled study of rei
... (truncated, 4 KB total)Resource ID:
966761554dbb1100 | Stable ID: MTQ0ZDFkMG