READY or Not: Reliable Enterprise Agent Deployment
Veronica Chatrath, Yuan Xue and a Scale AI team introduce READY, a framework that stops asking how well an agent performs and starts asking under what oversight policy and at what cost it can be deployed at a required reliability level.
Ask this paper
Deployment is a different question than benchmark success: Enterprise deployment asks whether an agent meets a reliability target under acceptable human oversight at tolerable cost. Benchmarks answer none of those three.
Qualification procedure, not a score: Given an agent, a workflow and a class of candidate oversight policies, READY measures the reliability and operating cost of the combined human-AI system, selects the minimum-cost policy meeting the target, and statistically qualifies it on held-out cases.
The headline case study: On a clinical audit spanning 16 agent systems and 750 cases, two systems separated by 0.3 points of autonomous accuracy (72.8 vs 72.5 percent) needed 39.2 percent versus 29.6 percent human review to qualify at the same 76 percent reliability target.
Open testbed: Workflow specification, execution, evaluation and qualification are decoupled and run on existing agent-evaluation infrastructure.
Why it matters: That 0.3-point example is the whole argument. Leaderboard-adjacent agents can differ by a third in the human labor they require, and no current benchmark shows you that.
Abstract
An AI agent can perform well on benchmarks and still be unsuitable for deployment. Existing AI-agent benchmarks measure whether an agent can complete realistic professional work, whereas enterprise deployment asks a different question: whether an agent can meet a required reliability level, under acceptable human oversight, and at tolerable cost. We introduce Reliable Enterprise Agent Deployment (READY), a framework for qualifying AI agents for deployment on enterprise workflows. READY preserves each workflow's own definition of successful execution while applying a common qualification procedure. Given an agent, a workflow, and a class of candidate oversight policies, READY measures the reliability and operating cost of the human-AI system, selects the minimum-cost policy that satisfies a specified reliability target, and statistically qualifies it on held-out cases. The resulting deployment profile characterizes the supported operating point: reliability, human-oversight burden, and cost. READY is implemented as an open testbed that decouples workflow specification, execution, evaluation, and qualification, and runs on existing agent-evaluation infrastructure. In an end-to-end clinical-audit case study spanning 16 agent systems and 750 cases, READY reveals differences hidden by autonomous performance: two systems separated by only 0.3 points in autonomous accuracy (72.8% vs. 72.5%) require 39.2% versus 29.6% human review, respectively, to qualify at the same 76% reliability target under the evaluated oversight policy. READY thus shifts enterprise agent evaluation from how well can the agent perform the work? to under what conditions, and at what cost, can it be reliably deployed? By making those conditions explicit and statistically testable, READY provides a basis for comparing agent systems, setting oversight requirements, and making evidence-based deployment decisions.