GAUGE: When Not to Trust LLM-as-a-Judge in User-Simulated Evaluation of Task-Oriented Agents
Umesh Bodhwani, Thanh Tran and Kai Wei (Amazon) check whether the common offline evaluation gate for task agents, an LLM user simulator plus an LLM judge, ranks agents the same way a verifiable task reward does.
Ask this paper
Scale: 25 agents from six providers, four judges, tau2-bench and SimulatorArena, about 3,700 transcripts.
Satisfaction versus success: 57.5% of conversations rated satisfied failed the customer's task (rho = -0.147), and this holds across five rater populations and both benchmarks.
Close pairs: The gate's overall ranking is robust (rho = 0.94), but on near-equal agents it promotes the lower-reward agent on 31% of pairs, compared with under 1% on pairs far apart.
Self-preference: Judges from the same model family as the agent add about +0.75 on a 7-point scale.
Recipe: A judge-free completion bit detects truncation regressions at no cost, and out-of-sample recalibration of the judge does not transfer.
Abstract
Comparing and selecting task-oriented LLM agents increasingly relies on a low-cost offline evaluation gate: persona-driven LLM user-simulators converse with each candidate, an LLM-as-a-judge scores the transcripts, and the higher-scoring agent is promoted. We introduce GAUGE, a reusable offline protocol that measures whether this gate's ranking matches a grounded verifiable reward across 25 agents from six providers on the $\tau^2$-bench and SimulatorArena benchmarks, separating two kinds of evaluation validity that release practices conflate: ranking validity and construct validity. First, a satisfaction-success gap: satisfaction carries essentially no information about task success, as conversations rated satisfied by our blind panel are decorrelated from actual success, with 57.5% of them failing the customer's task, a pattern consistent across five rater populations, both benchmarks, and every subjective dimension we rated. Second, while the gate's ranking is robust across the broad capability span, it loses resolution among the near-equal strong agents: this decision-disagreement rate jumps from $<$1% on wide-reward pairs to 31% on close pairs. The gate is thus human-validated yet mis-anchored. As a remedy, we propose a calibrate-then-trust cadence in which a judge-free completion bit is a zero-cost tripwire for truncation regressions.