Not All AI Agents Are Equal: Characterizing Resource and Performance Dynamics
Wonmi Choi and colleagues characterize the resource dynamics of LLM agents across retrieval-augmented QA, web search and software coding, and use the measurements to build two scheduling optimizations.
Ask this paper
The same tool behaves differently by task. Resource dynamics vary widely with what the agent is doing, so serving policies keyed on tool identity alone misallocate.
Concurrency exposes task-dependent bottlenecks. Running multiple requests together surfaces CPU, disk I/O and memory limits that single-request profiling does not show.
Faster model responses do not always help. Neither do more CPU cores, because the binding constraint shifts between local tool containers and remote API latency.
Two optimizations follow from that. CPU-aware tool admission and task-aware CPU allocation improve CPU-sensitive task latency about 5.4 times and cut average latency across tasks about 32 percent against native agents.
Abstract
LLM-based AI agents process user requests through iterative reasoning and tool execution, often involving the invocation of remote LLM APIs with local tool containers. This execution model can make the optimization of agent serving difficult because latency, local resource demand, and container bottlenecks inter-mix across requests. However, the current agent ecosystem runs without much consideration of resource dynamics, which results in significant waste of the precious resources. This paper analyzes the resource inter-mix of AI agents for three representative tasks: retrieval-augmented question answering, web search, and software coding. To this end, we characterize the latency with respect to the resource dynamics of processing multiple requests and tasks concurrently. Our measurements show that agents have a wide range of behaviors depending on tasks, so that even the same tool can differ substantially in resource dynamics. We also find that running multiple requests concurrently exposes task-dependent bottlenecks in resource dynamics such as CPU, disk I/O, and memory. Furthermore, we uncover that faster LLM responses or more CPU cores do not always accelerate agents. Based on these observations, we demonstrate new optimization opportunities that exploit the resource dynamics of tasks: CPU-aware tool admission and task-aware CPU allocation. Our results show that the latency of CPU-sensitive agent tasks improves $\sim$5.4$\times$, and the average latency across multiple tasks is reduced $\sim$32% compared to native agents.