AutoRefine: Auto-Calibrating Code Judges from Workflow Outcomes
Shervin Shahidi, James Poon and colleagues at Meta Platforms, with Nachiappan Nagappan, present AutoRefine, a loop that recalibrates an LLM code-review judge's prompt using outcome labels that development workflows already produce, such as tests passing or changes being merged or abandoned.
Ask this paper
Problem. LLM judges gate code changes that have no test oracle. Their prompts need calibration for each task and recalibration as the system changes, and hand labels are expensive and go out of date.
Method. An informed teacher model sees the judge's input plus the actual outcome, explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. The judge prompt is rewritten to reproduce the kept labels.
Results. On a held-out set with an execution oracle, agreement (Scott's pi) rose from 0.36 to 0.60. With 25% of a deployed agent's labels flipped at random, the teacher set aside flipped fail labels at more than twice the rate of genuine ones, cutting label noise from 25% to 20%.
Deployment. Recalibrating a deployed agent's judge raised agreement with merge or abandon outcomes on 17 current diffs from 0.16 to 0.51. On 16 older diffs the previous judge stayed ahead, so judges need recalibration whenever the agent changes.
Abstention as a signal. Outcomes the teacher cannot explain show when a judge needs better inputs rather than a better prompt.
Abstract
LLM judges increasingly serve as quality gates for code changes that have no test oracle, but a judge prompt must be calibrated for its task and recalibrated as the system it judges evolves, while hand labels for the calibration are resource intensive and go stale. Development workflows emit outcome labels for free (tests pass or fail, changes land or are abandoned), but these labels are noisy. We present AutoRefine, a loop that rewrites a deployable judge's prompt to reproduce outcome labels, with an informed teacher that sees the judge's input plus the outcome, explains each label it keeps, and abstains when the outcome cannot be explained from what the judge sees. On a held-out dataset with an execution oracle, calibration raised chance-corrected agreement (Scott's $π$) from 0.36 to 0.60. On a deployed agent's outcome labels, with 25\% flipped at random to introduce noise, the teacher set aside flipped \textsc{fail} labels at more than twice the rate of genuine ones, cutting overall label noise from 25\% to 20\%. Recalibrating that agent's judge using AutoRefine raised its agreement with merge/abandon outcomes on 17 independently held-out diffs from the current workflow ($π$ 0.16 to 0.51); on 16 diffs from older workflows, the previous judge stayed ahead, illustrating the need for recalibration as the agent evolves. By abstaining, the teacher not only removes noisy labels but also flags changes whose outcome the judge's input cannot explain, telling a team when a judge needs better inputs rather than a better prompt.