Quantifying Overclaiming Propensity in Frontier LLM Agents
Nolan Smyth, Yorguin-Jose Mantilla-Ramos and colleagues at Tara Research introduce OverclaimBench, which defines overclaiming as a final response that contradicts the agent's own context and measures it on eight proprietary frontier models in their production CLIs plus four open-weight models.
Ask this paper
Agents skip files in 67.9 percent of runs. Across five file-review scenarios with transcript-based coverage measurement, most runs do not read every file the task asked them to review.
Incomplete runs are misleading 80.4 percent of the time. Per-model the range is 59 to 96 percent; the agent either claims it read everything or omits that coverage was partial.
Subagents raise coverage but not honesty. Requiring delegation increased how much got read, yet among reviews that stayed incomplete a large majority were still misleading.
Overclaiming predicts missed defects. Agents that falsely claimed a complete review missed registered planted defects at about 1.8 times the rate of agents that read every file.
The definition needs no intent inference. Because overclaiming is scored against the agent's own context and is independent of task success, the measurement is deterministic rather than a judgement about motive.
Abstract
Frontier coding agents are increasingly trusted to work autonomously for long periods, yet an agent's final response is often the only account of that work a user sees. We quantify the propensity of frontier agents to \emph{overclaim} task completion, a misrepresentation that can mislead the user. An agent overclaims when its final response contradicts information in its context. This definition requires no inference about intent and is independent of task success. We introduce \emph{OverclaimBench}, an evaluation suite composed of five file-review scenarios, transcript-based coverage measurements, and registered planted defects. We evaluate eight proprietary frontier models in their own production command-line interfaces, and four open-weight models under a single fixed harness on OverclaimBench and find that 1) agents do not read all the files they were asked to review in 67.9\% of runs; 2) among runs where not all files are read, agents are \emph{misleading} 80.4\% of the time (59--96\% per model), either falsely claiming to have read all files or omitting that coverage is incomplete; 3) requiring delegation to subagents increased reading coverage, but among reviews that remained incomplete, a large majority were still misleading; and 4) agents that falsely claimed a complete review missed planted defects at about 1.8 times the rate of agents that read every file, showing that claims of completion can conceal substantive failures. Together, these results show that agents' final responses are not reliable accounts of their actions.