Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?
Ankit Sonthalia (University of Tübingen), Seong Joon Oh (KAIST) and colleagues from the ELLIS Institute Tübingen and Ecole Polytechnique introduce BOTTLED, a benchmark that asks agents to turn an LLM capability into a cheap reusable artifact, such as a small model or a program, for a large unlabelled workload.
Ask this paper
Setup. Agents receive the whole workload with fixed time, compute and API budgets, and choose their own approach. They must decide how much to spend building the artifact versus applying it.
Zero-shot skill does not transfer. Across 10 models and 3 tasks, 48 of 60 bottling runs score below the lower bound of their own model's zero-shot 95% CI, and 31 of 60 underperform the stronger of two small-model distillation baselines at the same token budget.
Similar models diverge. On ESCI, GLM 5.3 and GPT 5.6 Terra have similar zero-shot macro-F1 (0.593 and 0.573), but after bottling GLM keeps about 84% and GPT 5.6 Terra about 46%.
Savings are real. On query-product relevance, Opus 5 keeps about 82% of its zero-shot macro-F1 at roughly 657x lower cost, and about 94% of Jev's macro-F1 at a quarter of Jev's projected cost.
Leaderboard. Opus 5 leads with an aggregate relative gain of about 42%, followed by GLM 5.3 Flash and Gemini 3.1 Pro at about 39%.
Abstract
Large language models (LLMs) can solve many narrow tasks, but querying them separately for millions of related instances can be prohibitively expensive. Can LLM agents autonomously create cheaper solutions for such workloads? We call this ability "bottling": the ability to turn general capabilities into task-specific solutions that balance answer quality and amortised cost. We introduce BOTTLED, a benchmark in which agents receive an entire unlabelled workload and must complete it under fixed time, compute and LLM API budgets. Agents choose their own approach, such as training a small model or writing a reusable program. Across ten models and three tasks, we find that strong zero-shot task performance does not reliably translate into strong bottling capabilities. Models with similar zero-shot scores can differ substantially after bottling, and 48 of 60 bottling runs score below the lower bound of the 95% confidence interval of their model's zero-shot performance. Moreover, 31 of 60 runs underperform the stronger of two small-model distillation baselines with the same token budget. Nevertheless, bottling can yield substantial savings: on query-product relevance classification, Opus 5 retains about 82% of its zero-shot macro-F1 at roughly 657 times lower reported cost. Bottling is also competitive with Jev, a "system one" model built especially for cheap, repetitive inference: Opus 5 on the same task recovers about 94% of Jev's macro-F1 at a quarter of Jev's projected full-workload cost. BOTTLED provides a basis for evaluating and improving agents' ability to invest limited resources in reusable solutions for large, repetitive workloads.