ServerBench: How can your coding agents design specialized LLM serving systems?
Xinrui Zhong, Beidi Chen and colleagues at Carnegie Mellon University introduce ServerBench, which asks coding agents to design complete, specialized LLM serving systems and scores them on token value per GPU dollar.
Ask this paper
Objective. Agents optimize "dollars per dollar": latency-qualified token value per dollar of GPU cost. They design the server and choose how many concurrent application sessions to run.
Evaluation protocol. Stateful replay of real coding sessions preserves shared context and dependent requests, and an independent verifier checks the submitted system for correctness.
Headline result. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 reaches 2.97 dollars per dollar on Gemma E4B, 4.87x the vLLM reference.
Delivery rate. 22 of 33 runs produce verified systems and 13 exceed the strongest baseline, so most of the gap between models is in delivering a correct system.
Findings. High-value designs pair capacity improvements with load selection, good agents design experiments around workload characteristics, and reliable delivery requires verifying the new execution paths that optimizations introduce.
Abstract
Large language model (LLM) coding agents develop software through experimentation, offering an opportunity to design the serving systems supporting their own operation. Effective serving requires coordinating implementation and operating decisions under session dependencies, latency constraints, and correctness requirements. We introduce ServerBench, a benchmark for agentic design of complete, specialized LLM serving systems. Agents experiment against a common objective, dollars per dollar: latency-qualified token value per dollar of GPU cost. Stateful replay preserves session dependencies, and independent verification checks the submitted system. Across 11 coding-agent models and three deployments, Claude Sonnet 5.5 achieves 2.97 dollars per dollar on Gemma E4B, $4.87\times$ the vLLM reference; 22 of 33 runs yield verified systems, and 13 exceed the strongest baseline. Our analysis identifies three findings: high-value designs coordinate capacity improvements with load selection; agents interpret performance feedback through workload-relevant experiments; and reliable delivery requires verification of the execution paths introduced by optimization. Overall, ServerBench supports systematic study of how agents translate service objectives and feedback into effective, verified serving systems for given task specifications.