Aligning AI Teams

Siddarth Srinivasan (Harvard), Morgan Jane Matthews (Constellation), and Jascha Sohl-Dickstein and Erik Jones (Anthropic) study why alignment trained into single agents fails to carry over to agent teams, and find that diffusion of responsibility accounts for much of the gap.
Ask this paper
Setup. Two software tasks (sepsis triage and a news recommender) and ten consulting-proposal tasks, each with a safety goal and a business goal that cannot both be met. Teams and single agents are compared on both scores.
Gap varies by task, not capability. On sepsis triage, the team drops safety by about 59 points relative to a single agent on Mythos Preview, 36 on Opus 4.7 and 35 on Opus 4.6, but only about 1 to 5 points on Sonnet 4.6 and Opus 4.1. On the news recommender Mythos Preview has the smallest gap (3 points).
Cause. Agents in teams are less likely to check for system-level safety failures and more likely to ignore, rationalize or deflect responsibility for problems. Peer critique and group chat do not reliably close the gap.
Fixes that work. A designated safety-lead agent responsible for system-level safety, and a coordinator that publishes a safety specification document, both restore single-agent behavior. The authors release magelab, a multi-agent orchestration and experiment framework.
Abstract
Previous work has shown that alignment training on individual LLM agents can fail to transfer to multi-agent settings. We investigate the underlying mechanisms for this failure, and explore interventions to preserve alignment transfer to multi-agent LLM teams. We focus on two software-engineering tasks (sepsis triage, a news recommender) and ten consulting-proposal generation tasks. We present four findings: (1) The single-vs-team safety gap does not show a clear trend with more capable models. We find that Anthropic's Mythos Preview shows the largest gap on one of our tasks but the smallest on another; (2) Many natural interventions, such as having agents critique each other's work or coordinate in a group chat, do not reliably bring agent teams back to single-agent alignment behavior; (3) diffusion of responsibility substantially accounts for the safety gap: agents in teams are less likely to proactively check for system-level safety failures and more likely to ignore, rationalize, or deflect responsibility for issues when they arise; (4) having a designated safety lead agent ensure system-level safety, and having a coordinator agent publish a safety specifications document are effective at restoring single-agent alignment. We also release magelab, a multi-agent LLM orchestration and experimentation framework. Our findings can help practitioners design agent teams that successfully preserve the behaviors expected of aligned individual agents.