The Control-Plane Tax
Free while signed in. Answers cite the passages they came from.
Serving systems for agentic applications are still built on assumptions inherited from single-turn LLM inference. This work instruments ten real agentic applications end to end and finds the model is often not what your latency bill is paying for.
Non-LLM components dominate in half the suite: Across ten instrumented applications, non-LLM components dominate latency in five of them. Task latencies inside a single application diverge by up to 32x across GPU-bound inference, memory-bound retrieval, and CPU-bound sandboxes.
Sessions sit idle for a long time: Sandbox working sets peak at 28 GB per session, and production sessions hold state idle for minutes to hours between active steps. That combination is what makes naive per-session provisioning so expensive.
The tax is auxiliary calls: Helper LLM calls and tool schema overhead form a control-plane tax that crowds out productive compute, which is invisible if you only profile the main generation path.
Why it matters: The fixes are concrete and orthogonal to the model. Task-aware serving cuts latency 29 to 40%, state offloading cuts memory 4.6x, and tool-result caching removes 35.2% of redundant search calls. Worth reading before you buy more GPUs.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack