Measuring the Marginal Value of Verifier-Based RL Beyond Supervised Fine-Tuning
Amit Sagtani and Saloni Garg, independent researchers in Santa Clara, measure what RL from verifiable rewards adds on top of supervised fine-tuning in a controlled, preregistered study on a 1,877-task debugging benchmark.
Ask this paper
Benchmark. gpu-train-debug has 1,877 synthetic GPU and LLM training-failure diagnosis tasks across 30 archetypes, graded by a programmatic verifier with partial credit.
Pilot. 1,200 gold demonstrations nearly solved it (SFT 0.785 on the hard split), while 15 GRPO steps from the base model left it unchanged (0.314 vs 0.311).
Main result. At 60 GRPO steps after SFT, the in-distribution gain is +0.22 points (95% CI -0.06 to +0.51) and the out-of-distribution gain is +0.33 (CI +0.02 to +0.68).
Attribution. Comparing RL against the base model overstates its contribution. On the OOD split, demonstrations account for 51.5 of the 51.8 points the full pipeline gains.
Cause. The errors SFT leaves are mostly systematic and identical across samples, which gives GRPO little within-group variance to learn from. The whole study cost $8.69 in compute.
Abstract
Reinforcement learning from verifiable rewards (RLVR) is usually evaluated as an alternative to supervised fine-tuning (SFT), or as a stage that follows it, with the implicit assumption that the RL stage contributes gains of its own. We measure that contribution directly, in a controlled setting where both stages can be run to completion and compared exactly. We built gpu-train-debug, a deterministic benchmark of 1,877 synthetic GPU and LLM training failure diagnosis tasks across 30 archetypes, graded by a programmatic verifier with partial credit, and calibrated so that a mid-tier model scores in a 0.60 to 0.75 band. A pilot produced a surprise: 1,200 gold demonstrations nearly solved the benchmark (SFT 0.785 on the hard split), while 15 steps of GRPO from the base model did not move it (0.314 vs 0.311 base). We therefore amended the study, in writing and before the full run, to ask the sharper question: what does verifier-based RL add beyond demonstrations? At the pre-registered scale (60 GRPO steps, learning rate 1e-6), the answer is: no detectable improvement in-distribution (+0.22 points, 95% CI [-0.06, +0.51]), and a modest but measurable improvement under distribution shift (+0.33 points on a held-out OOD split, CI [+0.02, +0.68]). SFT accounted for most of the available improvement, while verifier-based RL produced only modest additional gains under distribution shift. The residual errors SFT leaves are mostly systematic, identical across samples, which likely reduces the within-group learning signal available to GRPO. A full 60-step cold-start GRPO arm confirms the pilot at the full horizon. The whole study cost $8.69 of compute and is fully reproducible from archived generations, frozen configs, and tagged code.