Evaluating Confidence-Gated Retrieval with Matched Trajectory Replay
Prateek Chhikara introduces matched trajectory replay, a protocol that holds answer states, evidence, budgets, and action costs fixed so confidence-to-action mappings in retrieval agents can be compared on their trajectory-level consequences rather than in isolation.
Ask this paper
Calibration is evaluated as an action policy: Raw verbalized confidence versus post-hoc isotonic calibration, compared across Mistral, GPT, and Qwen on HotpotQA and MuSiQue with everything else held fixed.
Large gains among committed answers: At the same numerical threshold, calibration raises accuracy among committed answers by up to 41 points across all six model-dataset pairs.
But overall accuracy can fall: Up to 15 points better on HotpotQA and up to 17 points worse on MuSiQue. The effect is a shift to a more selective, lower-risk operating point, not better answers.
Calibration does not price another retrieval: A calibration map fitted before retrieval helps through depths one and two but is worse than raw confidence at depth three for all three models. Deciding whether to retrieve again needs a separate value-of-information estimate.
Why it matters: Anyone gating retrieval or escalation on a confidence threshold is implicitly assuming calibration predicts benefit. It predicts risk, and the paper separates the two.
Abstract
Interactive language-model agents use confidence signals to decide whether to answer immediately, retrieve additional evidence (from memory or external knowledge), or defer. Yet confidence is usually evaluated in isolation, without measuring the trajectory-level consequences of the actions it triggers. We propose matched trajectory replay, a controlled protocol for comparing confidence-to-action mappings. The protocol holds candidate answer states, evidence points, budgets, and action costs fixed. We use it to compare raw verbalized confidence with post-hoc isotonic calibration in a multi-hop question-answering system using Mistral, GPT, and Qwen models on HotpotQA and MuSiQue datasets. At the same numerical commitment threshold, calibration changes which questions agents ultimately commit to answering. Across all six model-dataset pairs, it increases accuracy among committed answers by up to 41 percentage points. However, it can reduce coverage and increase retrieval use. Overall accuracy improves by up to 15 percentage points on HotpotQA but falls by up to 17 percentage points on MuSiQue. These effects reflect a shift to a more selective, lower-risk operating point, not improved answers or confidence ranking. A calibration map fitted before retrieval improves held-out calibration through retrieval depths one and two, but is worse than raw confidence at depth three for all three models. Additional evidence helps on average, but this aggregate effect does not establish whether confidence identifies which individual episodes will benefit from another retrieval. Taken together, these results show that calibration can make commitment risk interpretable, but it does not estimate the expected benefit of another retrieval. Retrieval therefore requires a separate value-of-information or utility estimate. Evaluations should report held-out calibration, risk-coverage, and retrieval cost.