ERPBench: Evaluating LLM Agents for Enterprise Decision-Making Across Competitive Market Ecologies
Xinran Zhang and colleagues at GAIR ask whether enterprise-agent rankings survive a change in who the agent is competing against, and find they largely do not.
Ask this paper
The design. The same 100 fixed problems are run in two matched market ecologies: Solo, where each agent competes against fixed rule-based opponents, and Arena, where six evaluated agents compete in a shared market.
Scale. Six model families, 1,200 model-level trajectories, 7,200 decision rounds in a six-round ERP simulation with coupled pricing, production, procurement, inventory, finance, and shared-market competition.
The rankings do not transfer. DeepSeek leads in Solo (252.29M mean valuation, mean rank 1.67) while Gemini leads in Arena (263.95M, 1.76). The two ecologies agree on the task-level winner for only 21 of 100 problems.
A concrete reversal. Gemini's bottom-rank rate falls from 22% in Solo to 0% in Arena.
Why it matters for evaluation design. Any agent benchmark with fixed scripted opponents is measuring performance against those opponents, and this quantifies how far that is from a competitive setting.
Abstract
Large language model (LLM) agents are increasingly proposed for enterprise workflows, yet existing evaluations rarely test whether business-decision conclusions transfer across competitive market ecologies. We introduce ERPBench, an execution-instrumented benchmark for enterprise decision agents in a six-round Enterprise Resource Planning (ERP) simulation with coupled pricing, production, procurement, inventory, finance, and shared-market competition. ERPBench evaluates the same 100 fixed problems in two matched competitive market ecologies: Solo, where each evaluated LLM agent competes against fixed rule-based opponents, and Arena, where six evaluated LLM agents compete in a shared market. Across six model families, this yields 1,200 model-level trajectories spanning 7,200 decision rounds. Under the observed service configuration, the leading model differs between ecologies: DeepSeek leads in Solo (252.29M mean valuation; mean rank 1.67), whereas Gemini leads in Arena (263.95M; 1.76). The two ecologies identify the same task-level winner on only 21 of 100 problems, and Gemini's bottom-rank rate falls from 22 % to 0 % in Arena. ERPBench supports paired evaluation of whether enterprise-agent rankings transfer across competitive market ecologies, supplemented by aggregate execution-intervention analysis. Code and benchmark resources are available in our https://github.com/GAIR-NLP/erp-bench.