Towards Robust Mathematical Reasoning
Free while signed in. Answers cite the passages they came from.
Google DeepMind introduces IMO-Bench, a comprehensive suite of benchmarks vetted by IMO medalists targeting International Mathematical Olympiad-level reasoning, featuring 400 diverse Olympiad problems with verifiable answers, 60 proof-writing problems with detailed grading schemes, and 1000 human-graded proofs, playing a crucial role in achieving historic gold-level performance at IMO 2025. - **Three-benchmark suite**: IMO-AnswerBench (400 robustified problems across Algebra, Combinatorics, Geometry, Number Theory at 4 difficulty levels), IMO-ProofBench (60 proof-writing problems with 4-tier grading), and IMO-GradingBench (1000 human-evaluated solutions for automatic grader development). - **Robustification prevents memorization**: Problems undergo paraphrasing, reformulation, numerical changes, and distractor addition to ensure models demonstrate genuine reasoning rather than pattern matching from training data. - **AnswerAutoGrader near-perfect accuracy**: Built on Gemini 2.5 Pro achieving 98.9% accuracy, handling semantic equivalence across different expressions (e.g., "all real numbers except -4" vs "(-∞,-4)∪(-4,∞)"). - **Historic IMO 2025 gold performance**: Gemini Deep Think achieved 80.0% on AnswerBench (+6.9% over Grok 4, +19.2% over DeepSeek R1) and 65.7% on advanced ProofBench (+42.4% over Grok 4 heavy). Strong novel problem results (61.1%) indicate genuine capabilities. - **ProofAutoGrader validation**: Achieves 0.96 (basic) and 0.93 (advanced) Pearson correlation with human experts across 14 public models. Systematic errors remain: score overestimation, missing logical errors, excessive penalties for unconventional solutions. - **Benchmark difficulty confirmed**: Combinatorics hardest (<50% for most models), GPT-5 only 65.6% on AnswerBench. Correct short answers don't guarantee sound reasoning, highlighting substantial room for advancement.
Three-benchmark suite: IMO-AnswerBench (400 robustified problems across Algebra, Combinatorics, Geometry, and Number Theory at 4 difficulty levels), IMO-ProofBench (60 proof-writing problems with 4-tier grading), and IMO-GradingBench (1000 human-evaluated solutions for automatic grader development).
Robustification prevents memorization: Problems undergo paraphrasing, reformulation, numerical changes, and distractor addition to ensure models demonstrate genuine reasoning rather than pattern matching from training data.
AnswerAutoGrader near-perfect accuracy: Built on Gemini 2.5 Pro, achieving 98.9% accuracy, handling semantic equivalence across different expressions (e.g., “all real numbers except -4” vs “(-∞,-4)∪(-4,∞)”).
Historic IMO 2025 gold performance: Gemini Deep Think achieved 80.0% on AnswerBench (+6.9% over Grok 4, +19.2% over DeepSeek R1) and 65.7% on advanced ProofBench (+42.4% over Grok 4 heavy). Strong novel problem results (61.1%) indicate genuine capabilities.
ProofAutoGrader validation: Achieves 0.96 (basic) and 0.93 (advanced) Pearson correlation with human experts across 14 public models. Systematic errors remain: score overestimation, missing logical errors, and excessive penalties for unconventional solutions.
Benchmark difficulty confirmed: Combinatorics hardest (<50% for most models), GPT-5 only 65.6% on AnswerBench. Correct short answers don’t guarantee sound reasoning, highlighting substantial room for advancement.
Abstract
Finding the right north-star metrics is highly critical for advancing the mathematical reasoning capabilities of foundation models, especially given that existing evaluations are either too easy or only focus on getting correct short answers. To address these issues, we present IMO-Bench, a suite of advanced reasoning benchmarks, vetted by a panel of top specialists and that specifically targets the level of the International Mathematical Olympiad (IMO), the most prestigious venue for young mathematicians. IMO-AnswerBench first tests models on 400 diverse Olympiad problems with verifiable short answers. IMO-Proof Bench is the next-level evaluation for proof-writing capabilities, which includes both basic and advanced IMO level problems as well as detailed grading guidelines to facilitate automatic grading. These benchmarks played a crucial role in our historic achievement of the gold-level performance at IMO 2025 with Gemini Deep Think (Luong and Lockhart, 2025). Our model achieved 80.0% on IMO-AnswerBench and 65.7% on the advanced IMO-Proof Bench, surpassing the best non-Gemini models by large margins of 6.9% and 42.4% respectively. We also showed that autograders built with Gemini reasoning correlate well with human evaluations and construct IMO-GradingBench, with 1000 human gradings on proofs, to enable further progress in automatic evaluation of long-form answers. We hope that IMO-Bench will help the community towards advancing robust mathematical reasoning and release it at this https URL.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack