Ctrl-Z: Controlling AI Agents via Resampling - arXiv

Abstract

Control evaluations measure whether monitoring and security protocols for AI systems prevent intentionally subversive AI models from causing harm. Our work presents the first control evaluation performed in an agent environment. We construct BashBench, a dataset of 257 challenging multi-step system administration tasks, and evaluate whether various safety measures can prevent an adversarially constructed AI agent from covertly downloading and executing malicious code in this environment. This multi-step setting introduces new attack and defense dynamics, which we investigate in order to design novel control protocols that prevent safety failures without hindering the ability of non-malicious agents to perform useful work. We introduce a class of control protocols called resample protocols that dynamically take additional samples of certain actions. We find these protocols significantly improve on existing techniques by selectively blocking the AI agent from executing suspicious code and incriminating the agent by generating additional examples of dangerous behavior. We measure the tradeoff between attack prevention and usefulness; our best protocol combines resampling with analysis of previous steps, reducing the success rate of attacks from 58% to 7% at a 5% cost to the performance of a non-malicious agent.

Summary

This paper introduces Ctrl-Z, a method for controlling AI agents by leveraging resampling techniques to maintain safety and alignment during agentic task execution. The approach aims to detect and correct undesired agent behaviors by resampling actions or trajectories that deviate from intended outcomes, providing a practical mechanism for runtime oversight of autonomous AI systems.

Key Points

• •Proposes resampling as a control mechanism for AI agents, allowing unsafe or misaligned actions to be detected and replaced during execution
• •Addresses the challenge of maintaining human oversight over AI agents operating across extended, multi-step tasks
• •Contributes to the AI control paradigm by offering a scalable method that doesn't require full interpretability of agent internals
• •Evaluates the effectiveness of resampling-based control against adversarial or misaligned agent behaviors
• •Relevant to deployment of agentic AI systems where real-time correction of dangerous actions is critical