Efficient Agents
Free while signed in. Answers cite the passages they came from.
This paper presents Efficient Agents, a new agent framework that achieves a strong efficiency-effectiveness balance in LLM-driven systems. The authors perform the first systematic study of agent design choices through the lens of economic efficiency, specifically, the cost-of-pass metric, which captures the expected monetary cost of solving a task. Their proposed agent retains 96.7% of OWL’s performance on the GAIA benchmark while cutting costs by 28.4%.
Backbone matters most for performance, but cost varies dramatically. Claude 3.7 Sonnet achieves top accuracy (61.8%) but at a 3.6× higher cost-of-pass than GPT-4.1. Sparse MoE models like Qwen3-30B-A3B are much cheaper but trade off effectiveness, indicating they may suit simpler tasks where efficiency is prioritized.
Test-time scaling yields diminishing returns. Using Best-of-N sampling improves accuracy only marginally (from 53.3% to 53.9% when N increases from 1 to 4), while cost-of-pass worsens significantly (0.98 → 1.28), suggesting naive scaling is inefficient.
Planning depth boosts performance, but not indefinitely. Increasing max steps from 4 to 8 raises accuracy from 41.8% to 52.7%, but going to 12 yields little additional benefit while increasing costs sharply.
Simpler memory works best. Surprisingly, retaining just observations and actions (“Simple Memory”) is both more effective and efficient than fancier memory designs like hybrid or summarized memory. It improves accuracy (56.4%) and reduces cost-of-pass (0.74) compared to a no-memory baseline.
Web browsing should be kept minimal. Broader search sources and basic static crawling provide the best trade-off. Heavy interactive browsing adds token bloat without accuracy gains.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack