How to Speculate about Uncertainty in Agentic Coding? A Draft-Model Gate Method
Konstantin Grotov and Valentin Malykh derive a failure-prediction signal for a black-box coding agent from its output tokens alone, by running a small draft model over the agent's already-generated trajectory in one forward pass.
Ask this paper
Speculative decoding, inverted. Instead of proposing tokens, the draft model scores tokens the agent already produced. No logits, weights, activations, or repeated sampling from the agent are needed.
Phase-aware features. Reasoning spans and action spans are separated before the speculative cross-likelihoods are turned into features and calibrated against a verifiable objective.
Deployed as a pre-execution veto. On Qwen3-Coder-480B and Claude 3.5 Sonnet, gating actions before execution cut execution error rate by 6 to 8 points and token cost by 14 to 19%.
Transfers without retraining. The signal held on out-of-distribution benchmarks and across agent models, which matters because the draft model never sees the agent's internals.
General interface. The output is a failure-likelihood score that routing, human escalation, or extra test-time compute can each consume.
Abstract
LLM agents deployed for software engineering fail expensively: they act confidently wrong, and bad actions are recognized only after costly execution and retry. We present Speculative Uncertainty (SU), a method that recovers a predictive failure signal for a black-box agent from its output tokens alone, with no access to logits, weights, activations, or repeated sampling. Inverting speculative decoding, a small open-weight draft model scores the agent's already-generated trajectory in a single forward pass. From these speculative cross-likelihoods we extract phase-aware features by separating the reasoning and action spans, and calibrate them against a verifiable objective. SU produces a failure-likelihood score that any downstream policy, such as routing, human intervention, or extra test-time compute, can consume directly. To show the signal is actionable, we instantiate one such policy, a pre-execution veto gate, on software engineering agents Qwen3-Coder-480B and closed-source Claude 3.5 Sonnet, cutting execution error rate by 6-8 percentage points and token cost by 14-19% in deployment, transferring to out-of-distribution benchmarks without retraining, and generalizing across agent models.