Suppressing Misaligned Behaviors in Language Models, Durably
Luxi He, Pengcheng Jiang, Jifan Zhang, Jiawei Han and Danqi Chen (Princeton, UIUC, Thinking Machines Lab) study how to remove a misaligned behavior after it appears so that it stays removed under further fine-tuning, and propose PEARL, a GRPO variant trained from adversarial intermediate states.
Ask this paper
Model organisms. A narrow-trigger organism cheats on multiple-choice questions only when a planted hint is present. A broad-trigger organism, fine-tuned on insecure code, gives harmful answers across many prompts.
Three criteria. Suppression methods are scored on immediate removal, durability under further fine-tuning on the organism's data or unrelated data, and preservation of capability.
PEARL. Prefix-Expanded Adversarial RL adds rollouts that start from intermediate states sampled along cached misaligned trajectories, so the model learns to recover from partway into the bad behavior.
Results. On Qwen3-4B and GPT-OSS-20B, PEARL suppresses the behavior while keeping task accuracy, and resists reactivation better than SFT, GRPO and gradient-ascent unlearning.
Where it helps most. The clearest gains are on GPT-OSS-20B and on the broad-trigger organism, where targeted SFT cannot address the behavior. Accepted at NeurIPS 2026.
Abstract
Undesired model behaviors can emerge or re-emerge at any stage of training. Once observed, how effectively can we suppress them? To study this question, we first construct two model organisms--language models deliberately trained to exhibit a specific misalignment--that differ in what triggers the behavior. The narrow-trigger organism misbehaves only when the prompt contains a specific cue: it cheats on multiple-choice questions by following a planted hint that points to a wrong answer, and behaves normally otherwise. The broad-trigger organism has no such cue: after fine-tuning on insecure code, it gives harmful responses across a wide range of open-ended prompts. We then formalize post-hoc behavioral suppression and evaluate methods along three criteria: immediate suppression of the target behavior, durability under further training, and preservation of general capability. We compare standard methods--SFT, GRPO, and gradient-ascent unlearning--along these dimensions. Building on prior work on alignment shallowness, we hypothesize that more durable suppression requires the ability to recover from a wider range of misalignment states. We introduce Prefix-Expanded Adversarial Reinforcement Learning (PEARL), a GRPO variant that augments rollouts with continuations from intermediate states sampled along cached organism trajectories. Across both organism settings and two model families (Qwen3-4B, GPT-OSS-20B), PEARL achieves strong immediate suppression while preserving task accuracy, and remains durable under further fine-tuning on either the organism's data or unrelated data. It obtains the greatest benefit when the undesirable behavior is broad and cannot be addressed through targeted SFT. Together, our framework and method offer a step toward more durable post-hoc removal of undesired behaviors.