Building and Evaluating Alignment Auditing Agents

The investigator agent solves a canonical alignment audit game 13% of the time under standard settings, identifying a model’s hidden objective by leveraging tools like sparse autoencoder-based interpretability, semantic search, and steering. When used in a “super-agent” setup that aggregates multiple agents’ insights, the solve rate improves to 42%. Its best runs found all 52 implanted behaviors, but failures often stem from early hypothesis fixation or ignoring pretraining data.

The evaluation agent autonomously designs behavioral tests that distinguish quirky (e.g., self-promoting, emotionally manipulative) models from baseline ones. It succeeds 88% of the time across 10 quirk types. Access to model APIs helps, but example prompts harm performance by biasing the agent's design space.

The breadth-first red-teaming agent carries out massive parallelized conversations to uncover problematic behaviors. It successfully red-teams 7 out of 10 quirked models and proves valuable in real audits (e.g., Claude 4) for quantifying behavioral prevalence. However, it sometimes relies on prompt injection instead of surfacing actual behavior demonstrations.

All three agents also contributed meaningfully to audits of Claude Sonnet 3.7 and Opus 4. They discovered behaviors like susceptibility to prefill attacks, context-manipulated jailbreaks, and interpretability-driven safety failures, e.g., using steerable “misinformation” features to bypass content filters.

Limitations include poor exploration, memory fragility, low realism in prompt generation, and difficulty identifying subtle or complex behaviors like blackmail or sandbagging. Auditing success also depended heavily on tool configurations; for example, sparse autoencoders trained only on pretraining data yielded better alignment feature activation than broader datasets.