Single-Query Black-Box Calibration Auditing via Logit Bias
Roman Plaud and colleagues at Institut Polytechnique de Paris, Onepoint and Ghent University show that a logit_bias parameter is enough to recover exact probability thresholds from an API that hides output probabilities, using one query per sample.
Ask this paper
The obstacle: Standard calibration metrics need continuous output probabilities, and commercial providers increasingly do not expose them.
The mechanism: Any API exposing logit_bias can be manipulated mathematically to evaluate exact probability thresholds with strictly one query per sample, so the audit cost scales with the dataset rather than with the desired precision.
The estimator: A provably consistent estimator of the True Calibration Error for binary tasks, built on that thresholding mechanism.
What it enables: Calibration auditing of black-box foundation models used as zero-shot classifiers, which is where miscalibration turns directly into deployment risk.
Abstract
Evaluating the calibration of Large Language Models (LLMs) is critical for their safe deployment as zero-shot classifiers. Yet, commercial API providers increasingly hide the continuous output probabilities required by standard calibration metrics. To bypass this opacity, we demonstrate that any LLM API exposing a logit\_bias parameter can be mathematically manipulated to evaluate exact probability thresholds using strictly one query per sample. Leveraging this mechanism, we introduce a novel and provably consistent estimator of the True Calibration Error for binary tasks. Our approach therefore provides an efficient framework for auditing black-box foundation models.