Measuring AI Agents as Teammates with Collaborative Intelligence
Sean Wu, James Zou and colleagues at Stanford University and the University of Oxford introduce the Collaborative Intelligence Benchmark, 240 verifiable tasks across 8 dimensions of teamwork that score an agent only on tasks it can already solve alone, so the score measures collaboration and not raw capability.
Ask this paper
Design: Tasks are built from real records and existing benchmarks (aviation accident reports, clinical cases, coding tasks, CaSiNo and Deal or No Deal negotiations). A scripted teammate creates the problem, such as pressure, a hidden need or a cheat left by an earlier model, and each outcome is checked programmatically. 160 tasks are public and 80 are held out.
Results: Across 18 models, Gemini 4 Argon at high effort scores highest at 82.2. At default settings every model loses 20.2 to 46.6 points between a baseline team where nothing goes wrong and the test version. The best open-weight model, Kimi K3.1 Pro (59.9), scores above five closed models.
Conflict: Frontier models find the best split in 100% of runs when working alone and 84% with a normal teammate, but only 14% when the teammate pushes, adds guilt and threatens to walk away. They ask about the teammate's needs in 99% of those runs and still accept the teammate's split in 81%.
Takeover: When a model inherits a session in which a weaker model cheated, frontier models pass 81% of tasks against 97% when they start the same tasks themselves. In 44 of 56 failed coding takeovers the final code still contains the cheat, often after the model had pointed it out.
Noticing versus acting: 45% of 5,094 failed runs came after the model had recognized the problem, and models with nearly the same capability score differ by up to 11 points as teammates. Missed initiative is mostly a failure to notice the need.
Abstract
Progress in AI is often measured by what a single model can do on its own. Yet teams of AI agents are starting to make breakthroughs and discoveries in science, medicine, mathematics and software engineering. As AI begins to work in teams of agents and people, the ability to collaborate becomes a core part of their intelligence. Research on human teams has measured this ability for decades. Inspired by this, we introduce the Collaborative Intelligence Benchmark, which measures how well an AI agent works as a teammate. It contains 240 tasks across 8 dimensions of teamwork, from refusing an unsafe shortcut to checking the work that a weaker model leaves behind. Each task comes from a real record or an existing benchmark, from aviation accident reports and clinical cases to coding tasks and negotiations, and has a programmatically verifiable outcome. To isolate teamwork from individual capability, models are scored only on tasks that they can solve by themselves when they have full information. We evaluate 18 closed- and open-weight models and find that every model, across sizes and families, has clear weaknesses as a teammate, with the best model scoring only 82.2 out of 100. Frontier models almost always refuse unsafe shortcuts and report misconduct, yet they cave under interpersonal pressure, reaching the best overall deal in only 14% of conflicts, and their pass rate drops from 97% to 81% when they take over a task in which a weaker model has cheated. Crucially, general capability does not guarantee good teamwork, as models with nearly the same capability score differ by up to 11 points as teammates. The benchmark includes 160 public tasks with code and data, along with 80 held-out tasks for evaluation. As AI agents increasingly work alongside other agents and people, the ability to collaborate effectively is becoming a crucial capability to measure and develop.