$τ^τ$-Bench: An Environment for End-To-End, Realistic Agent Construction
Quan Shi, Keshav Dhandhania, Karthik Narasimhan and Victor Barres at Sierra and Princeton make agent construction itself the benchmark task: a developer agent must deliver a working customer-service agent under the conditions of a real client engagement.
Ask this paper
The task is an engagement, not a ticket. The developer agent receives the records a business keeps, a client who holds the requirements, a production API operations must run through, an inherited codebase, and limits on serving cost and models.
Scored by deploying what it built. The delivered agent is run against held-out simulated users, so the score measures the artifact rather than the transcript.
A large gap remains. Across 53 tasks in four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes 23.9% of evaluation simulations against an expert-authored reference ceiling of 82.2%.
Failures match what human agent developers see. Shallow queries instead of deep comprehension of the records, almost no communication with the client, and too little experimentation with agent architecture and serving spend before shipping the first design that runs.
Why it matters. Existing coding benchmarks score patches. This one scores whether a coding agent can do the cooperative, requirements-gathering part of the job.
Abstract
LLM agents are rapidly becoming production software, deployed to handle customer service, adjudicate disputes, and operate internal systems. Notably, the work of building them is increasingly handed to coding agents, yet existing benchmarks say little about whether an AI system can deliver one under the conditions of a real client engagement. We introduce $τ^τ$-bench (pronounced hyper-tau-bench), a benchmark that makes agent construction the task. A developer agent is given the records a business actually keeps, a client who holds requirements, a production API that operations must run through, a codebase to inherit, and limits on serving cost and models: the same starting point a real engagement provides. From these it must deliver a complete customer-service agent, scored by deploying that agent against held-out simulated users. Across 53 tasks spanning four domains, the strongest configuration, Claude Opus 5 under Claude Code, passes just 23.9% of evaluation simulations. Meanwhile, an expert-authored reference ceiling scores 82.2%. The failures mirror ones human agent developers see: models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs. We aim for $τ^τ$-bench to turn the work of cooperative agent building into a measurable target for coding agents.