Open Thoughts
Free while signed in. Answers cite the passages they came from.
This paper presents OpenThoughts3, a systematic recipe for curating supervised fine-tuning (SFT) data that advances the performance of open-source reasoning models. The authors develop OpenThinker3-7B, a 7B parameter model trained on their new 1.2M example dataset (OpenThoughts3-1.2M) derived from over 1,000 controlled experiments. Despite using no reinforcement learning, OpenThinker3-7B outperforms all other open-data 7B and 8B models on standard math, code, and science reasoning benchmarks, even beating models trained with larger-scale or mixed SFT+RL pipelines. Key insights and contributions:
Best-in-class 7B open model: OpenThinker3-7B achieves state-of-the-art results on AIME25 (53.3%), LiveCodeBench (51.7%), and GPQA Diamond (53.7%), outperforming DeepSeek-R1-Distill-Qwen-7B by 15–20 percentage points across tasks.
Scaling laws with clean design: The authors ablate every step in the data pipeline, question sourcing, filtering, teacher choice, deduplication, and answer sampling, showing how each incrementally lifts performance. For instance, using multiple answers per question (16×) improved results more than simply increasing question diversity.
QwQ-32B as a better teacher than stronger models: Surprisingly, QwQ-32B yielded better student models than DeepSeek-R1 or Phi-4 despite lower benchmark scores, suggesting teacher choice affects trace quality more than raw performance.
Filtering matters more than verification: Question filtering based on response length and LLM-estimated difficulty was more predictive of downstream gains than traditional heuristics (e.g., fastText) or even filtering based on correctness verification, which had negligible effects.
Data quality over diversity: Mixing only the top 1–2 question sources per domain consistently outperformed using many sources, indicating that question quality is more important than dataset heterogeneity.
Open-source impact: The full datasets and models are released at [openthoughts.ai](http://openthoughts.ai/), providing a reproducible benchmark for open reasoning research.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack