Interface-Induced Trajectory Censoring
Wenbo Wang (City University of Hong Kong) shows that a reported tool-call rate can be exactly zero while the model emits well-formed calls the whole time, because the serving adapter censors the trajectory before the executor or scorer ever sees it.
Ask this paper
Same weights, same seeds, opposite scores: on BFCL v4's own executor and scorer, holding weights, cases, decoding and seeds fixed and changing only the serving adapter, the same model scores 0.00 or 0.96.
A 2x2 locates it exactly: across chat template and parser, both main effects are exactly zero and the entire effect sits in the interaction, so no single component is defective and fixing one side buys precisely nothing.
It scales with the model, invisibly: across a 21x scale range of Qwen2.5-Coder the server parses 0 of 100 at every size while well-formed emitted calls rise to 80 of 100 at 32B, so the bigger the model the larger the hidden loss.
It reaches into training: in verl's AgentLoop at 7B, 45 of 115 generations carry a complete call, and 0 are accepted, 0 execute, 0 return an observation.
Repairing the adapter restores mechanism, not magic: parsing goes 0 to 84 and rescues 0 to 9 with pass rate 53 to 62, not significant, and the author releases a 98-line preflight check that catches every silent failure reported.
Abstract
Agent evaluations report a tool-call rate read off the serving stack. That number can be zero while the model is emitting well-formed calls: the interface censors the trajectory before anything downstream sees it. On BFCL v4's own data, executor and scorer, holding weights, cases, decoding and seeds fixed and changing only the serving adapter, the same model scores 0.00 or 0.96 / 0.19. A 2x2 over chat template and parser locates the effect exactly: both main effects are exactly zero and all of it sits in the interaction -- no component is defective, and repairing one side of the contract buys precisely nothing. On tau-bench's 115 interactive retail tasks the same swap moves server-parsed calls from 0 to 636 and tasks reaching any tool execution from 0 to 103. Our probe reproduces the funnel across a 21x scale range of Qwen2.5-Coder: the server parses 0/100 at every size while well-formed emitted calls rise to 80/100 at 32B (~72 after calibration against an adjudicated gold standard). Under a matched envelope, across a comparable scale span, the silent fraction stays at 0-2, a prediction committed to the repository before the run. Llama-3.1-8B's 23% rate of calling the task function itself as a tool falls to 0 under one strict:true flag. The mismatch reaches inside the training loop, and its consequence is scale-dependent: in verl's AgentLoop at 7B, 45 of 115 generations carry a complete call; 0 are accepted, 0 execute, 0 return an observation. At 1.5B the same zero is over-determined, so we report the two scales separately. At evaluation time, repairing the adapter restores the mechanism but not a significant outcome gain: parsing 0->84, rescues 0->9, pass rate 53->62 (n.s.). We release a 98-line preflight check that catches every silent failure here. The observed tool-call rate is not a property of the model alone; it is a property of the model-interface stack that measures it.