AutoTuneBench: Trustworthy Measurement for Agent Auto-Tuning of LLM Serving Engines
Li Chen (harnets.ai) presents AutoTuneBench, a benchmark and measurement protocol for LLM agents that tune GPU kernels and serving engines, built after a four-day pilot of 619 model calls showed that the propose-measure-keep loop produces untrustworthy speedups.
Ask this paper
Four failure modes: Strawman baselines create speedups, absolute times do not transfer across machines, saturated tasks make comparisons meaningless, and infrastructure defects look like results.
Protocol as code: The protocol is frozen with test-enforced provenance, a database validator rejects out-of-protocol results, anti-cheat checks run outside what the agent can modify, and comparisons use pre-registered readouts with a 5% cross-run CV cap.
Corrected numbers: The best kernel is 10.6x against a naive baseline but 2.03x against an honest one; one configuration gives 1.174x on one machine and 1.0049x on another.
KernelBench check: KernelBench Level-1 admits 51% of tasks whose median speedup over PyTorch eager is 1.0001x.
Abstract
Large language model agents tune GPU kernels and serving engines through a closed loop of propose, measure, and keep, but the measurements behind this loop are not trustworthy. We characterize four failure modes from a four-day pilot corpus of 619 model calls: strawman baselines manufacture speedups, absolute times do not transfer across machines, saturated tasks nullify comparisons, and infrastructure defects impersonate science. We present AutoTuneBench, a benchmark and measurement protocol that makes trust architectural. The protocol is frozen as code with test-enforced provenance; a database-level validator rejects out-of-protocol results; anti-cheat checks run outside the agent's modification surface; comparisons follow pre-registered readouts; and measurements anchor to externally published results, grounded in paired-seed statistics with a 5\% cross-run coefficient-of-variation cap. Honest measurement rewrites the headlines: our best kernel reads 10.6x against a naive baseline but 2.03x against the honest one; one configuration delivers 1.174x on one machine and 1.0049x on another; a pre-registered on/off comparison nulls at a shared wall (2.4840 vs 2.4957\,ms); and the KernelBench Level-1 suite admits 51\% of tasks with median speedup 1.0001x over PyTorch eager. The protocol, the two-engine corpus (vLLM and SGLang), and its audit trail are released as open artifacts.