SAVeR$^2$: Reasoning-based Safety Alignment for Large Reasoning Models via Verifiable Rewards
Yichen Sun, Zhixuan Chu and colleagues at Zhejiang University, Chongqing Ant Consumer Finance and Alibaba Group propose SAVeR², an RL safety-alignment method for large reasoning models that splits safety into four verifiable rewards and removes gradient components that conflict with reasoning.
Ask this paper
Objective. Safety for reasoning models is defined as non-malicious reasoning and answers, consistency between the two, and explicit identification of the user's intent.
Method. Training combines a supervised warm-up, difficulty-aware data selection, four fine-grained verifiable safety rewards, and a gradient projection that removes the part of the safety gradient that conflicts with the reasoning gradient.
Results. On DeepSeek-R1-Distill-Qwen-7B, the average across StrongReject, BSA, XSTest and WildJailbreak rises from 44.03 for the base model to 92.80, against 83.50 for the best baseline (PPO). At 14B it reaches 94.28 against 87.22 for STAR-1.
Over-refusal and capability. The paper reports low over-refusal on XSTest, FalseReject and OKTest and no loss on AIME24, MATH-500, LiveCodeBench, HumanEval and MMLU, with experiments from 1.5B up to 671B.
Abstract
Large Reasoning Models (LRMs) have achieved impressive performance via explicit Chain-of-Thought (CoT) reasoning, yet this process introduces critical safety risks. We formalize LRM safety objectives as requiring non-malicious reasoning and answers, safety consistency, and explicit intent identification. We argue that existing alignment methods suffer from poor adversarial generalization, a significant ``safety tax'' on reasoning, and unfaithful coarse-grained rewards, primarily due to their neglect of the LRM's intrinsic reasoning capabilities for safety. To address these, we propose \textbf{SAVeR$^2$}, a novel \textbf{S}afety \textbf{A}lignment method via \textbf{R}ewards with \textbf{R}easoning capability. SAVeR$^2$ decomposes the safety objective into four fine-grained verifiable reward signals and utilizes a difficulty-aware data selection strategy to stabilize training. Crucially, we introduce a reasoning-preserving gradient projection mechanism that analytically resolves conflicts between safety and reasoning gradients, effectively eliminating the safety tax. Extensive experiments on models ranging from 1.5B to 671B demonstrate that SAVeR$^2$ achieves superior safety performance and significantly lower over-refusal rates across multiple benchmarks without compromising general reasoning capabilities.