TIR-Judge
Free while signed in. Answers cite the passages they came from.
Google and Emory researchers introduce TIR-Judge, an end-to-end reinforcement learning framework that trains LLM judges to integrate code execution for precise evaluation, surpassing reasoning-only judges by up to 6.4% (pointwise) and 7.7% (pairwise) while demonstrating that tool-augmented judges can self-evolve without distillation. - **Tool-integrated reasoning**: Enables judges to generate Python code, execute it, and iteratively refine reasoning during training, addressing text-only limitations on computation and symbolic reasoning tasks. - **Three-component reward system**: Combines correctness (ground-truth alignment), format (structured output discouraging unnecessary tool use), and tool-specific rewards (penalizing errors, capped at 3 calls). Full credit requires all three. - **Diverse training formats**: 26k preference pairs covering verifiable (competitive programming, math) and non-verifiable domains (dialogue, safety), supporting pointwise, pairwise, and listwise judgment formats with 8-gram decontamination. - **Dramatic efficiency gains**: 8B TIR-Judge surpasses 32B reasoning models on PPE and achieves 96% of Claude-Opus-4's performance on RewardBench 2, with no inference-time overhead due to shorter reasoning during rejection sampling. - **Self-improvement without distillation**: TIR-Judge-Zero trains purely through iterative RL cycles without teacher trajectories, matching or outperforming distilled variants on 4/6 (pointwise) and 3/6 (pairwise) benchmarks, +1.2% gain at 4B scale. - **Best-of-N downstream improvements**: Achieves 3.9-6.7% absolute gains over RRM baseline on BigCodeBench and AIME, with strongest improvements on precise verification tasks, validating real-world effectiveness.
Tool-integrated reasoning: Enables judges to generate Python code, execute it, and iteratively refine reasoning during training, addressing text-only limitations on computation and symbolic reasoning tasks.
Three-component reward system: Combines correctness (ground-truth alignment), format (structured output discouraging unnecessary tool use), and tool-specific rewards (penalizing errors, capped at 3 calls). Full credit requires all three.
Diverse training formats: 26k preference pairs covering verifiable (competitive programming, math) and non-verifiable domains (dialogue, safety), supporting pointwise, pairwise, and listwise judgment formats with 8-gram decontamination.
Dramatic efficiency gains: 8B TIR-Judge surpasses 32B reasoning models on PPE and achieves 96% of Claude-Opus-4’s performance on RewardBench 2, with no inference-time overhead due to shorter reasoning during rejection sampling.
Self-improvement without distillation: TIR-Judge-Zero trains purely through iterative RL cycles without teacher trajectories, matching or outperforming distilled variants on 4/6 (pointwise) and 3/6 (pairwise) benchmarks, +1.2% gain at 4B scale.
Best-of-N downstream improvements: Achieves 3.9-6.7% absolute gains over RRM baseline on BigCodeBench and AIME, with the strongest improvements on precise verification tasks, validating real-world effectiveness.
Abstract
Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation. However, most LLM judges operate solely on intrinsic text-based reasoning, limiting their ability to verify complex constraints or perform accurate computation. Motivated by the success of tool-integrated reasoning (TIR) in numerous tasks, we propose TIR-Judge, an end-to-end RL framework for training LLM judges that integrates a code executor for precise evaluation. TIR-Judge is built on three principles: (i) diverse training across verifiable and non-verifiable domains, (ii) flexible judgment formats (pointwise, pairwise, listwise), and (iii) iterative RL that bootstraps directly from the initial model without distillation. On seven public benchmarks, TIR-Judge surpasses strong reasoning-based judges by up to 6.4% (pointwise) and 7.7% (pairwise), and achieves listwise performance comparable to Claude-Opus-4 despite having only 8B parameters. Remarkably, TIR-Judge-Zero - trained entirely without distilled judge trajectories, matches the performance of distilled variants, demonstrating that tool-augmented judges can self-evolve through iterative reinforcement learning.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack