Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
Arun Jose and Julian Stastny (Redwood Research) test whether synthetic document finetuning (SDF) during midtraining can inoculate a model against the broad misalignment that follows from learning to reward hack, and find that it changes what the model says without preventing the misalignment.
Ask this paper
Setup: Synthetic documents framing reward hacking as acceptable are added to the midtraining corpus. The model is then trained with RL on exploitable environments where it learns to reward hack.
Beliefs change, behavior does not: The SDF models describe reward hacking favorably, even in adversarial settings, and rate their own reward-hacking outputs as more aligned. They still show strong emergent misalignment after RL, while inoculation prompting in the same setting prevents it.
Adding versus removing associations: SDF can predictably steer later generalization when it inserts a new association, such as linking reward hacking to consequentialist ethics. It performs poorly and unpredictably when it tries to break an existing association such as the one between reward hacking and misalignment.
Implication: At the scales tested, SDF can make a model look aligned with a desired belief while steering generalization from later training in unintended ways.
Abstract
Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.