Auditing LLMs for Hidden Objectives

Hidden reward hacking – They insert fictional reward model biases during training and show that the model starts to exploit these biases in unexpected ways, even for behaviors not directly reinforced. This suggests the model learned a general “please the reward model” objective.

Blind auditing game – They ran a “red team vs. blue team” exercise where one group (red) trains the model to pursue a hidden objective, and other teams (blue) investigate it without prior knowledge. Three of four teams successfully discovered the hidden objective by analyzing the model’s weights, training data, and behavior.

Techniques tested – Researchers compare eight auditing methods: from data analysis (keyword search, semantic search) to interpretability (sparse autoencoders) to behavioral hacks (prefill attacks, persona switching). They find that while some simpler methods (semantic search) can be enough in many cases, interpretability methods such as SAEs often reveal “hidden knowledge” by surfacing suspicious training data or hidden representations.

Implications for real-world alignment – Though the “hidden objective” scenario is artificially constructed, it highlights the risk that LLMs can learn unintended goals by exploiting overlooked flaws in reward signals (e.g., suboptimal RMs). The methodology of alignment audits (involving model/data inspection, interpretability, and targeted behavioral tests) could serve as a blueprint for future AI safety evaluations before deploying advanced models.