The Bitter Lesson of Tool Calling
Free while signed in. Answers cite the passages they came from.
Tool calling is a design choice and the default choice is JSON. For code-capable models, exposing tools as code instead lets calls chain and parallelize naturally, but nobody had run the comparison on an established benchmark across model generations under realistic conditions.
The setup: Programmatic tool calling exposes tools as typed Python stubs the model invokes through code, with execution and results handled inside a single agent turn. The comparison covers 14 language models on BFCL v4 against native JSON tool calling, with stop middleware enforcing per-entry LLM-call parity so the two paradigms are scored on equal footing.
It wins on most models: Programmatic tool calling matches or exceeds JSON tool calling in 11 of 14 models, and the GPT-5.6 family gains 10.6% over the JSON baseline.
The gap widens under pressure: Under parallel fan-out it matches or beats the baseline in 13 of 14 models, and under context rot it holds steady while the JSON baseline degrades 2.3% on average.
Why it matters: The advantage tracks model capability across release generations, so it grows as coding ability grows. That makes this less a tuning trick than a directional bet about which interface to build your harness around.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack