Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff
Nathaniel Bottman at Incubilate and Alexei Oblomkov at UMass Amherst derive when RL with verifiable rewards trains a model to bluff, using rewards that pay b for an unverifiable verdict and r for a certificate a checker accepts.
Ask this paper
Threshold. To first order, a GRPO step makes the model more honest exactly when p r / b > 1 + kappa, where p is the rate the checker accepts the model's certificates and kappa is the pull from items where the unverifiable verdict is correct.
Honesty as a race. A checker lets p rise during training, so a run slightly below the threshold can still end honest. A lottery with the same mean payoff and no check never recovers from below the threshold.
Dip and ceiling. Kappa starts large and falls, so runs just above the threshold first dip toward bluffing, on code almost to zero, before recovering. Paying too much for the honest verdict spills over: at p r / b = 3, both code models call about half of correct programs buggy.
Schedule. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, while training at the lower payoff from the start falls to 4 to 15%.
Probe. When honesty collapses on depth-2 Datalog, a linear probe fit before RL still reads the truth, so the model represents what it no longer reports.
Abstract
Two rewards with the same expected payoff at the start of training can end a reinforcement-learning run in opposite places, with an honest model or one that bluffs. We study rewards that pay $b$ for a verdict that a checker cannot verify ("this program is correct") and $r$ for a certificate that it accepts (an input revealing a bug). To first order, a GRPO step makes the model more honest exactly when $\tilde{p}\,r/b>1+κ$. Here $\tilde{p}$ is the rate at which the checker accepts the model's certificates, weighted toward the items on which the model still gives both verdicts, and $κ$ is the pull of the items on which the unverifiable verdict is right. Both terms move during training. A checker raises $\tilde{p}$, so near the threshold honesty is a race, which a run just below the threshold can win and a run under a lottery (random pay with the same mean) cannot. The term $κ$ starts large and falls as the other items settle, so a run just above the threshold first dips toward bluffing, on code nearly to zero, before climbing back. Paying the honest verdict also pulls the other items toward it, which predicts a ceiling on the payoff ratio, above which the model gives the honest verdict where it is wrong. On propositional Datalog, two code models, and abstention on grade-school math, with $p$ the model's success rate before RL, no run above $pr/b=1$ ends collapsed, and runs under the checker can survive slightly below it. At $pr/b=3$, above the ceiling, both code models call about half of correct programs buggy. On depth-3 Datalog, training generously under the checker and then cutting the payoff keeps the model at least 67% honest in three seeds, at a ratio where the same model trained from the start falls to 4-15% and is at most 47% honest after 160 steps. On depth-2 Datalog, when honesty collapses, a linear probe fit before RL still reads the truth.