Rethinking Verbalized Confidence for LLM-as-a-Judge: A Compatibility Shift on Post-2025 Proprietary Models
Yu-Chung Hsiao (Cisco Systems) shows that on post-2025 proprietary models, verbalized confidence is a more robust soft score for LLM-as-a-Judge than log-probabilities, which reverses the standard advice.
Ask this paper
Scope: Experiments cover SummEval, AggreFact and HelpSteer2 with up to 18 LLMs, comparing verbalized confidence with logprob-based scoring such as G-Eval.
Compatibility shift: On post-2025 top-tier models verbalized confidence is the better signal. The change does not appear under accuracy-only reporting.
Two additions: An overconfidence advisory and self-debate improve calibration, score spread and robustness to task subjectivity.
Generation effect: Post-2025 models accept these additions with little balanced-accuracy cost, while pre-2025 models pay a measurable penalty.
Recommendation: Use soft scoring in LLM judges instead of defaulting to hard labels, and retest scoring choices when judge models change generation.
Abstract
Verbalized confidence, long dismissed as overconfident, coarse, and prone to round-number clustering, is now the more robust soft-scoring mechanism for LLM-as-a-Judge on top-tier proprietary models. Across SummEval, AggreFact, and HelpSteer2, spanning up to 18 LLMs, we show that the standard advice to prefer log-probabilities no longer holds on post-2025 models, where verbalized confidence is the better signal. We call this a compatibility shift. On top of a standard verbalized-confidence baseline, we introduce two new ingredients: an overconfidence advisory and self-debate. Together they improve calibration, score-distribution spread, and robustness to task subjectivity. We further observe a generation effect: post-2025 models accommodate these two additions with little balanced-accuracy cost, whereas pre-2025 models pay a measurable penalty. Compared with logprob-based G-Eval, verbalized confidence is the more subjectivity-robust soft signal on GPT-family top-tier releases. The shift is invisible under accuracy-only reporting. Rather than defaulting to hard predictions, we recommend broader use of soft scoring in LLM-as-a-Judge. More broadly, verbalized confidence has moved from a weaker substitute for logprobs to a practical soft-scoring mechanism for contemporary LLM judges.