Reasoning Traces Tailored for Small Models
Small models often get worse when you SFT them on long, high-quality CoT from big teachers. This paper pinpoints why and fixes it with Reverse Speculative Decoding (RSD): let the teacher propose tokens, but let the student approve them only if they are probable under the student. Result: traces that stay correct while matching the student’s distribution, which small models can actually learn from.
Ask this paper
Core idea: At each step, sample a teacher token and keep it only if the student assigns ≥ p_th probability, else fall back to the student’s own token. This filters high-surprisal spikes that small models cannot track, smoothing token-level difficulty without simplifying the logic.
Why it matters: Direct SFT of Qwen3-0.6B on s1K-1.1 traces hurts average accuracy by 20.5%. Training on RSD traces instead yields +4.9% average gains across AIME24, AIME25, GPQA-Diamond, MATH500.
Data recipe that works for tiny models: Use a tokenizer-compatible teacher (s1.1-7B) and student (Qwen3-0.6B). Generate RSD traces with rejection sampling; when a problem cannot be solved, salvage the first 128 tokens via UPFT-style prefix training. Despite only 180 full solutions and many prefixes, the 0.6B student improves, showing that distributional alignment beats volume.
Key diagnostic: The strongest failure predictor is the share of sub-1% tokens under the student. s1K-1.1 traces contain many such tokens and degrade learning; RSD cuts these to near zero.
Not universal, must be tailored: RSD traces are model-specific. Traces built with Qwen3-0.6B as the “approver” do not transfer to Qwen3-1.7B, Llama-3.2-1B, Gemma-3-1B, or Phi-4-Mini. Running RSD per target model helps, but repeated multi-step RSD on the same model degrades performance via distributional drift.