Calibration is the Bottleneck: An Action-Class Diagnostic of Multi-Turn Tool-Calling
Kangjia Zhao and colleagues at Zhejiang University and Om AI argue that aggregate multi-turn tool-calling accuracy hides which of two orthogonal failures a model actually has, and give a diagnostic that separates choosing the wrong action class from executing the right one badly.
Ask this paper
A four-class action space. Failures are decomposed over TOOL_CALL, ASK, REFUSE, and CONFIRM, which is where the interesting behavior lives once raw tool-call accuracy saturates.
A self-revealing bound. Accuracy is bounded by Gold Action Recall. Violation of the bound exposes state-grader masking of miscalibration; large slack localizes execution failure inside TOOL_CALL.
Heavily tool-trained families look better than they are. The diagnostic separates models whose standing is inflated by miscalibration from those that actually choose context-appropriate actions.
Calibration is fragile. A single context-only perturbation moves accuracy in opposite directions across model families, up to +11.5 against -21.0 points on the same scenario, and the direction depends on the perturbation mechanism.
The recommendation. Multi-turn tool-calling leaderboards should publish action-class diagnostics alongside aggregate accuracy, because the aggregate cannot see the dominant failure.
Abstract
Multi-turn tool calling is a core evaluation scenario for large language model (LLM) agents. On public tool-calling benchmarks, open-weight models now approach or even surpass closed-source frontier models in aggregate accuracy. However, this metric averages over many different multi-turn situations and obscures whether progress is balanced across them. We propose an action-class-oriented diagnostic framework that decomposes multi-turn failures into two orthogonal modes: action-class miscalibration and action-execution failure. The framework operates over a four-class action space (TOOL_CALL/ASK/REFUSE/CONFIRM) and introduces a self-revealing upper bound Acc <= GAR (Gold Action Recall); the two modes show up as bound violation (Acc > GAR, exposing state-grader masking of miscalibration) and large bound slack (GAR >> Acc, localizing execution failure within TOOL_CALL). We validate it on a panel of tool-calling models across multiple multi-turn benchmarks. Across our panel, the diagnostic reveals action-class miscalibration as a substantial failure mode the state grader cannot see. This gap inflates standing for heavily tool-trained families, which our diagnostic separates from families with context-appropriate action choice. Calibration is reshapable through context-only perturbations, but the reshape is heterogeneous: a single perturbation moves accuracy in opposite directions across families (up to +11.5 vs -21.0 pp on the same scenario), and its effect further depends on the perturbation mechanism. We argue that multi-turn tool-calling evaluations should supplement aggregate accuracy with action-class diagnostics that expose what the model actually does in each scenario.