TailSFT: Filtered Fine-Tuning Improves Post-Training Performance
Sadhika Malladi, Samy Jelassi, Dylan Foster, Jordan T. Ash and Akshay Krishnamurthy (Microsoft Research) ask whether standard SFT produces the model you actually want to run RL on, and propose a one-line change that says no.
Ask this paper
Filter out what the model already fits: TailSFT drops sequences the model has already fit during training, concentrating learning on the under-modeled tail of the data distribution. That is the entire modification.
Coverage is the target, not loss: The design follows from prior work showing pass@K and coverage predict post-RL performance. Standard SFT spends gradient on already-fit sequences and quietly narrows the distribution RL will later need to explore.
Validated on OLMo-3 7B: TailSFT often improves pass@16 on math and coding evaluations, with the filtering criteria justified through both controlled experiments and theoretical analysis rather than an ablation table alone.
Why it matters: The SFT-then-RL pipeline is nearly universal and the SFT half is usually treated as a solved commodity step. This says the commodity step is actively working against the stage that follows it.
Abstract
Reinforcement learning post-training drives reasoning and agentic capabilities in modern AI systems, yet a growing body of work shows that it is most effective when used to fine-tune an already capable base model. We question whether existing pipelines yield models that are most suitable for reinforcement learning. Building on prior work highlighting the role of coverage and pass@K as predictors of post-RL performance, we design a simple modification to supervised fine-tuning, TailSFT, which filters out already fit sequences during training, thereby focusing learning on under-modeled regions, or the tail, of the data distribution. We justify and validate the design choices in TailSFT, particularly the specific filtering criteria, through a combination of controlled experiments and theoretical analysis. On OLMo-3 7B, TailSFT often improves pass@16 performance on math and coding evaluations, with gains up to 17% absolute, while incurring minimal computational overhead. These higher-coverage checkpoints consistently translate to up to 4% absolute pass@1 gains in subsequent GRPO runs, demonstrating that TailSFT checkpoints serve as better initializations for RL. We further introduce a lightweight diagnostic for identifying settings where TailSFT is most likely to help. More broadly, our results motivate a principled, stage-aware approach to model development, in which intermediate checkpoints are judged by how effectively they support subsequent training.