Agent Architecture Cost-Success Trade-offs: A Controlled Evaluation across Enterprise-Style SOPs
Subhrangshu Nandi, Chirag Shah and colleagues at Amazon and the University of Washington hold the model, tasks and tools fixed and vary only the agent architecture across 13 enterprise standard operating procedures, comparing a single function-calling agent, a skill-augmented agent, centralized multi-agent orchestration and a triage swarm.
Ask this paper
Setup. Eight arms across four architecture families, plus a skill-definition quality rubric used to build matched high- and low-quality skill corpora.
No universal winner. Task success ranges from under 10% to near 100% depending on the procedure, and no architecture is statistically separable in aggregate.
Cost. Every coordinated arm spends 1.4-4.7x the tokens and 2.6-5.8x the latency of the single agent, usually for small and non-significant gains.
Skills. High-quality skills raise success to 73.1% from 67.4%, while degraded skills fall below the single-agent baseline.
Implementation quality. A plausible multi-agent build loses to the single agent, but a targeted redesign of it is the only arm that significantly beats the baseline (75.6% vs 67.4%).
Abstract
Enterprises increasingly use large language model agents to execute standard operating procedures (SOPs): long, branching, tool-intensive workflows. Yet there is little controlled evidence for choosing among competing agentic architectures. Holding the model, tasks, and tools fixed and varying only the architecture, we compare four designs (a function-calling single agent, a skill-augmented agent, centralized multi-agent orchestration, and a triage swarm; eight arms) across 13 enterprise SOPs, and introduce a skill-definition quality rubric to build matched high and low-quality skill corpora. Three takeaways emerge. (1)~\textbf{There is no universal best architecture; the right choice is SOP-dependent.} On a given SOP, task success ranges from under $10\%$ on the hardest to near $100\%$ on the easiest; no architecture is statistically separable in aggregate, and added structure only plausibly pays off on complex, branching SOPs (a directional, not yet significant, trend). (2)~\textbf{Added architecture cost must be justified by a task-success gain.} Every coordinated arm (multi-agent or skill-augmented) spends $1.4$--$4.7\times$ the tokens and $2.6$--$5.8\times$ the latency of the single agent for gains that are usually small and rarely significant, making the single agent the efficient default; high-quality skills do lift success (to $73.1\%$ from $67.4\%$) while degraded skills fall below it. (3)~\textbf{Implementation quality outweighs the single- versus multi-agent choice.} A plausible multi-agent implementation loses to the single agent, yet a targeted redesign of it becomes the only arm to significantly beat the baseline ($75.6\%$ vs.\ $67.4\%$).