Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
Hoang Cuong Nguyen, Mark Dras and Usman Naseem at Macquarie University compare supervised fine-tuning, reasoning-augmented fine-tuning and ORPO across Llama-3.1-8B, Gemma-2-9B and Qwen3-8B, and find that the choice of post-training method, not only the safety data, determines how refusal is computed inside the model.
Ask this paper
Training method reshapes internal structure: reasoning-augmented training produces a distinct kind of refusal computation that appears in all three models, so the mechanism is method-driven rather than architecture-driven.
Architecture still matters independently: it separately shapes internal structure and how reliably refusal can be steered, so the two factors have to be reported together.
No method achieves all three desirable properties: refusal that is not concentrated in a few fragile components, safety gains that do not cost general capability, and safety behavior correctable through small targeted edits. Every method studied gives up at least one.
Direct implication for red-teaming: the concentration of refusal in few components is what makes steering attacks work, so a method that spreads refusal is worth more than one that scores higher on a static refusal benchmark.
Code and models released at github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.
Abstract
How do the methods used to train language models to refuse harmful requests shape how that refusal actually works inside the model? We compare three post-training methods - supervised fine-tuning, reasoning-augmented fine-tuning (training on reasoning chains that justify a safety decision), and preference optimization (ORPO) - across three architecturally distinct models (Llama-3.1-8B, Gemma-2-9B, Qwen3-8B). We find that training method, not just data, reshapes how refusal is computed internally: reasoning-augmented training consistently produces a distinct kind of refusal computation, visible across all three models, while architecture independently shapes internal structure and how reliably refusal can be steered. Most importantly, no method we study achieves all three properties we would want from safe alignment at once: refusal that isn't concentrated in a few fragile components, safety gains that don't cost general capability, and safety behavior correctable through small, targeted edits. We caution against treating current post-training methods as a solved, reliable defense, especially for security-critical use. Code and models are available in https://github.com/hoangcuongnguyen2001/Beyond-Shallow-Alignment.