AUDITPLAN: Commit, Then Answer for Auditable Safety Alignment
Sai Sri Pushpa Jampani, Kshitij Mishra and Asif Ekbal make a model emit a machine-checkable safety plan before answering, and reward the answer only when that plan is correct.
Ask this paper
Plan then answer, in one model. The model first emits a compact structured plan recording a threat label, intended action, and explicit constraints, then answers conditioned on it; the plan is hidden from users at deployment but available to auditors.
FAITHGATE. Reinforcement learning grants answer reward only when the safety plan is correct, which removes the incentive for safe-looking rationales that do not constrain the answer.
Results on Qwen2.5-3B-Instruct. Attack success rate falls from 24.0% to 11.6%, leakage success rate from 1.0% to 0.36%, and over-refusal from 11.0% to 2.0%, beating answer-only RL, free-form explanation, and weighted-sum structured rewards.
Holds across sizes. Qwen2.5-1.5B-Instruct shows the same trend, and confirmation runs on Qwen-3-4B-Instruct and Qwen2.5-7B-Instruct preserve it.
Significance. Blanket refusal and unfaithful rationale are two different failures that final-answer grading cannot tell apart; committing to a structured plan first makes both measurable.
Abstract
Safety tuning pipelines judge only the final answer, which makes it difficult to distinguish robust refusal from two undesirable shortcuts: blanket refusal on benign requests and polished but unfaithful safety rationales that do not actually constrain the answer. We propose AUDITPLAN, a single-model plan-then-answer approach where the model first emits a compact structured safety plan and then answers conditioned on it. The plan records a threat label, intended action, and explicit constraints, enabling machine-checkable auditing while remaining hidden from users at deployment. We train this behavior with supervised fine-tuning followed by reinforcement learning with FAITHGATE, a reward-gating objective that grants answer reward only when the safety plan is correct. This discourages safe-looking but unfaithful behavior and promotes tighter plan-answer coupling. Across Qwen backbones, AUDITPLAN improves both robustness and auditability: on Qwen2.5-3B-Instruct, FAITHGATE reduces ASR from 24.0% to 11.6%, LSR from 1.0% to 0.36%, and over-refusal from 11.0% to 2.0%, outperforming answer-only RL, free-form explanation, and weighted-sum structured rewards. Similar trends hold for Qwen2.5-1.5B-Instruct. Larger-model confirmation runs on Qwen-3-4B-Instruct and Qwen2.5-7B-Instruct preserve the same trend suggesting that explicit internal commitments can make safety alignment more faithful, robust, and auditable.