Magistral
Free while signed in. Answers cite the passages they came from.
Mistral introduces Magistral, its first reasoning-focused LLM line, alongside a custom RL training stack that enables pure reinforcement learning from scratch. In contrast to prior approaches that rely on distillation from teacher models, Magistral trains directly using online RL with text-only data and custom reward shaping. The work yields two open models: Magistral Medium (based on Mistral Medium 3) and the open-sourced Magistral Small (24B), which is bootstrapped via SFT on Medium’s outputs, followed by RL. Key insights:
Pure RL can rival distillation: Magistral Medium was trained without any reasoning traces and achieved a 50% boost in AIME-24 (pass@1) over the base model. Across reasoning benchmarks like LiveCodeBench and GPQA, it performs on par or better than DeepSeek-R1 despite lacking a distillation phase.
Multilingual and multimodal generalization emerge: Reinforcement learning on textual math/code data surprisingly enhances multimodal reasoning and preserves instruction-following and tool-calling capabilities. Multilingual reasoning is achieved via simple reward shaping, enforcing the same-language output and chain-of-thought.
Efficient async infrastructure: An asynchronous RL setup with NCCL weight broadcasting enables generators to roll out sequences continuously, receiving weight updates without blocking. This helps maintain on-policyness while maximizing GPU utilization.
Reward shaping and CoT format enforcement: Rewards are conditioned on correct formatting (<thinktags, boxed answers, markdown blocks), correctness (via SymPy and test cases), length penalties, and language consistency (using fastText). Failure to meet formatting results in an immediate zero reward.
Training heuristics and ablations: Analysis shows that reward and performance scale logarithmically with output length. Longer completions correlate with higher reward. RL moves weights in a low-dimensional space, as shown by PCA of checkpoints. Ablations on batch size, advantage normalization, and entropy (via ε ᵢg ) highlight stability tradeoffs.
Open-source release: Magistral Small (24B) is released under Apache 2.0 and performs competitively under three setups, SFT-only, RL-only, and SFT+RL, with the final combo yielding the best results across math and code tasks.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack