Budget Aware Test-time Scaling
Free while signed in. Answers cite the passages they came from.
Researchers discover that simply expanding tool-call budgets without proper awareness fails to improve agent performance. They introduce BATS, a framework that makes web search agents budget-aware, enabling more strategic resource allocation and pushing the cost-performance Pareto frontier. - **Key finding:** Increasing token budgets improves LLM performance, but expanding tool-call budgets without awareness yields no improvement. Resource consciousness is essential for effective agent scaling. - **Budget Tracker Plugin:** A lightweight mechanism that provides agents with continuous awareness of remaining resources, enabling strategic decision-making throughout task execution. - **BATS framework:** Dynamically adjusts exploration strategy based on remaining capacity - deciding whether to pursue promising leads deeper or explore alternative paths. - **Results:** Budget-aware approaches produce more favorable scaling curves and systematically improve agent efficiency under computational constraints. First comprehensive study of budget-constrained tool-augmented agents.
Key finding: Increasing token budgets improves LLM performance, but expanding tool-call budgets without awareness yields no improvement. Resource consciousness is essential for effective agent scaling.
Budget Tracker Plugin: A lightweight mechanism that provides agents with continuous awareness of remaining resources, enabling strategic decision-making throughout task execution.
BATS framework: Dynamically adjusts exploration strategy based on remaining capacity - deciding whether to pursue promising leads deeper or explore alternative paths.
Results: Budget-aware approaches produce more favorable scaling curves and systematically improve agent efficiency under computational constraints. First comprehensive study of budget-constrained tool-augmented agents.
Abstract
Scaling test-time computation improves performance across different tasks on large language models (LLMs), which has also been extended to tool-augmented agents. For these agents, scaling involves not only "thinking" in tokens but also "acting" via tool calls. The number of tool calls directly bounds the agent's interaction with the external environment. However, we find that simply granting agents a larger tool-call budget fails to improve performance, as they lack "budget awareness" and quickly hit a performance ceiling. To address this, we study how to scale such agents effectively under explicit tool-call budgets, focusing on web search agents. We first introduce the Budget Tracker, a lightweight plug-in that provides the agent with continuous budget awareness, enabling simple yet effective scaling. We further develop BATS (Budget Aware Test-time Scaling), an advanced framework that leverages this awareness to dynamically adapt its planning and verification strategy, deciding whether to "dig deeper" on a promising lead or "pivot" to new paths based on remaining resources. To analyze cost-performance scaling in a controlled manner, we formalize a unified cost metric that jointly accounts for token and tool consumption. We provide the first systematic study on budget-constrained agents, showing that budget-aware methods produce more favorable scaling curves and push the cost-performance Pareto frontier. Our work offers empirical insights toward a more transparent and principled understanding of scaling in tool-augmented agents.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack