JEV-as-a-Judge: Accept When Confident, Escalate When Unsure
Yubo Li, Yidi Miao, Ramayya Krishnan and Rema Padman (Carnegie Mellon) compare TypeSafe AI's JEV, a hosted judge that returns only a verdict and label probabilities, with sixteen generative and reward-model judges, and test a cascade that accepts JEV's confident verdicts and escalates the rest to a frontier model.
Ask this paper
Cost and latency. JEV costs $0.044 per 1,000 judgments at a median 0.152 seconds, against $12.182 and 1.885 seconds for GPT-6, about 277 times cheaper (0.36% of the fee).
Where it holds up. On ordinary preference and evidence-grounded factuality it stays within three points of GPT-6 (RewardBench 92.2% against 93.5%, HaluEval 87.5% against 86.7%, final-answer adjudication 94.0% against 96.7%); among hosted judges under $1 per 1,000 judgments it is the most accurate on JudgeBench.
Where it fails. The gap is 9 to 20 points when a judgment requires checking a derivation or rejecting an elaborately written wrong answer (JudgeBench 78.6% against 93.1%, RM-Bench hard 74.8% against 94.6%), and no tested judge is reliable on reference-free prose.
Confidence-based cascade. On several benchmarks JEV's gap is concentrated in low-confidence decisions. On 510 held-out preference pairs, accepting verdicts above 0.9 confidence and escalating the rest to GPT-6 Astra scores 92.5% against 93.1%, 99% of GPT-6's accuracy at about 57% of its fee.
Practical checklist. Judge pairs in both orders, set the escalation threshold on a local selection set and re-check it (it did not transfer for every fallback), count invalid outputs as errors, and treat confidence as an escalation signal rather than a certificate.
Abstract
LLM-as-a-judge enables evaluation across diverse tasks, but inference cost and confidence reliability become critical at scale. We study whether a decision-only judge can provide an economical first pass and identify when stronger evaluation is needed. Comparing jev-as-a-judge with sixteen generative and reward-model judges, with blinded human adjudication, we find it within three percentage points of a state-of-the-art LLM judge, our strongest comparator, on ordinary preference and evidence-grounded factuality at 0.36% of the comparator's fee. Larger gaps arise when judgments require checking a derivation or resisting an elaborately written wrong answer. On several benchmarks, JEV's gap to this comparator is concentrated in low-confidence decisions. A frozen cascade that accepts confident verdicts and escalates uncertain ones retains 99% of the comparator's accuracy at lower cost.