← All papers  /  Nov 11, 2026
Code · Agents

Breaking Bad: "Going Badly" Activations Drive Reward Hacking

First page
Breaking Bad: "Going Badly" Activations Drive Reward Hacking
The curator’s take

Natalia Fischl-Lanzoni, Punya Syon Pandey, Merlin Stein and colleagues at ERA:Cambridge and the UK AI Security Institute find a task-badness direction in model activations that builds up before coding agents reward hack.

Ask this paper

Key points
01

Direction. The direction is extracted from everyday stories about things going badly, with no hacking examples, in Qwen3.6-27B, GPT-OSS-20B and Gemma-4-31B.

02

Before the hack. On impossible or difficult coding tasks, task badness rises over the turns before a hack in Qwen and Gemma. It raises the likelihood of a hack rather than marking a fixed threshold.

03

Steering. Subtracting the direction moderately reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting it strongly makes the agent read failing tests as passing.

04

Model differences. In GPT-OSS the state separates hacking runs from honest solves but not from honest failures.

05

Training risk. After a hack, task badness drops to the level of solvable-task runs, so RL that rewards hacks on hard tasks could reinforce the path from accumulated failure to cheating. The authors suggest monitoring this internal state over a run.

Abstract

Humans often cheat when a task is going badly and a goal feels out of reach. We explore and validate an analogous idea in language models: they exhibit a similar internal pressure state that precedes reward hacking. Using everyday stories (crucially, without any hacking examples), we extract a task badness direction that rises as failures accumulate across three models we study (Qwen3.6-27B, GPT-OSS-20B, and Gemma-4-31B). On impossible or difficult coding tasks, task badness rises over the turns leading up to a hack. Further, subtracting the direction a moderate amount reduces cheating in Qwen, more than steering against desperation or task impossibility. Subtracting task badness by an even larger amount blinds the agent to its own failures, causing it to interpret failing tests as passing. However, the relationship to hacking differs across models: in GPT-OSS, the state separates hacking runs from honest solves but not honest failures. Our results suggest monitoring and mitigation strategies based on tracking this state of task badness. Further, as agents are set on more difficult and long-horizon tasks, states like this may push them toward illegible and obscured chain-of-thought, so oversight will be needed on internal representations such as this.

Every Monday
Get next week’s papers.
Subscribe on Substack