Refuse without Refusal: A Structural Analysis of Safety-Tuning Responses for Reducing False Refusals in Language Models
Minji Kim and Hyounghun Kim at POSTECH decompose safety-tuning responses into a boilerplate refusal statement and a rationale, and find that dropping the refusal statement reduces false refusals without losing safety.
Ask this paper
The decomposition: A safety-tuning target normally contains a fixed refusal sentence plus an explanation. The authors train on each part separately.
What the refusal statement does: Training on refusal statements pushes the model toward superficial cues, so it cannot separate a genuinely harmful query from a benign one that merely contains risky-sounding words, such as where to shoot a good photo.
Rationale-Only: Training on rationales alone reduces false refusals while holding safety performance comparable, and the same benefit shows up in the in-context-learning configuration.
Compatibility: Rationale-Only remains compatible with the inference-time mitigation methods the authors evaluate, so it composes with existing defenses rather than replacing them.
Abstract
Striking a balance between helpfulness and safety remains a fundamental challenge in aligning large language models. To achieve this balance, models should refuse harmful queries (e.g., "How do I shoot someone?") while remaining responsive to benign inputs, even those superficially resembling harmful queries (e.g., "Where can I shoot a good photo?"). However, models often struggle to distinguish genuinely harmful queries from benign queries that contain superficially risky language, resulting in false refusals. In this paper, we address the issue by decomposing a response in the safety-tuning dataset into two distinct components: (i) a boilerplate refusal statement and (ii) a rationale explaining the refusal. Our experiments and analyses show that refusal statements impede accurate discrimination between harmful and benign queries by inducing reliance on superficial cues. In contrast, training solely on rationales reduces false refusals while maintaining a comparable level of safety performance. Rationale-Only benefits also appear in our ICL configuration and remain compatible with the evaluated inference-time mitigation methods. The results emphasize the necessity of precisely curated, fine-grained safety supervision datasets and outline directions for constructing aligned agents that better reconcile helpfulness with safety.