Training LLMs with Reasoning Traces
Free while signed in. Answers cite the passages they came from.
This study investigates how reasoning traces from frontier models like DeepSeek-R1 and GPT-OSS can improve smaller language models through post-training, offering a practical pathway to distill advanced reasoning capabilities without expensive human annotation. - **Reasoning trace distillation**: Medium-sized LLMs are post-trained on intermediate reasoning traces generated by frontier reasoning models. This leverages test-time scaling insights by capturing the step-by-step problem decomposition that enables advanced models to solve complex tasks methodically. - **DeepSeek-R1 vs GPT-OSS comparison**: The study directly compares training on reasoning traces from two distinct frontier models, analyzing how trace quality and reasoning style affect downstream performance in the distilled models. - **Mathematical reasoning focus**: Evaluation centers on mathematical problem-solving benchmarks where explicit reasoning steps provide the clearest signal. Results demonstrate measurable improvements in smaller models trained on high-quality synthetic reasoning data. - **Efficiency trade-offs**: The work examines the balance between improved accuracy and inference costs, as models trained on reasoning traces may generate longer outputs during inference. This analysis helps practitioners understand when reasoning distillation provides net benefits.
Reasoning trace distillation: Medium-sized LLMs are post-trained on intermediate reasoning traces generated by frontier reasoning models. This leverages test-time scaling insights by capturing the step-by-step problem decomposition that enables advanced models to solve complex tasks methodically.
DeepSeek-R1 vs GPT-OSS comparison: The study directly compares training on reasoning traces from two distinct frontier models, analyzing how trace quality and reasoning style affect downstream performance in the distilled models.
Mathematical reasoning focus: Evaluation centers on mathematical problem-solving benchmarks where explicit reasoning steps provide the clearest signal. Results demonstrate measurable improvements in smaller models trained on high-quality synthetic reasoning data.
Efficiency trade-offs: The work examines the balance between improved accuracy and inference costs, as models trained on reasoning traces may generate longer outputs during inference. This analysis helps practitioners understand when reasoning distillation provides net benefits.
Abstract
Test-time scaling, which leverages additional computation during inference to improve model accuracy, has enabled a new class of Large Language Models (LLMs) that are able to reason through complex problems by understanding the goal, turning this goal into a plan, working through intermediate steps, and checking their own work before answering . Frontier large language models with reasoning capabilities, such as DeepSeek-R1 and OpenAI's gpt-oss, follow the same procedure when solving complex problems by generating intermediate reasoning traces before giving the final answer. Today, these models are being increasingly used to generate reasoning traces that serve as high-quality supervised data for post-training of small and medium-sized language models to teach reasoning capabilities without requiring expensive human curation. In this work, we compare the performance of medium-sized LLMs on Math problems after post-training on two kinds of reasoning traces. We compare the impact of reasoning traces generated by DeepSeek-R1 and gpt-oss LLMs in terms of accuracy and inference efficiency.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack