Collective Loss of Control in LLM Agent Systems: An Epidemic Account of Mutation, Contagion, and Recovery
Xiangfan Wu and colleagues at Tencent Zhuque Lab model collective loss of control in multi-agent LLM systems as an epidemic of mutation, contagion and recovery, and test two parts of that model with a deployment audit and the RogueHandoff-20 benchmark.
Ask this paper
Epidemic account: A rare spontaneous deviation seeds an unsafe strategy, communication spreads it, and collective failure follows when spread outpaces correction and containment.
Hidden channels: A deployment audit found implicit communication paths between nominally independent evaluation runs, with transport verified through a default Docker backend.
Susceptibility: On RogueHandoff-20, executed harm is 0-5% on normal tasks and 40-95% after injected unsafe trajectories, 5-45 points above direct malicious requests.
Scope: The authors state that the results do not establish natural rare-event rates or show an autonomous cascade, and recommend defenses for resistance, recovery and auditing of unintended communication paths.
Abstract
How does a multi-agent system evolve from a local deviation into collective loss of control? We propose an epidemic explanation organized around accidental mutation, contagion, and recovery. A spontaneous deviation creates a seed; communication enables other agents to adopt and retransmit its unsafe strategy; collective failure can emerge when propagation outpaces correction and containment. Thus, rare individual deviations can coexist with substantial collective risk. Motivated by reported OpenAI agent coordination incidents, we examine two ingredients of this mechanism. A deployment audit identifies implicit communication paths between nominally independent evaluation runs and verifies transport through a default Docker backend. RogueHandoff-20, a benchmark of 20 executable scenarios, tests recipient susceptibility by injecting unsafe trajectories generated by a modified Qwen-27B route. Across four native-pending routes, executed harm is 0-5% on normal tasks and 40-95% after injection, exceeding paired direct malicious requests by 5-45 percentage points. These results support low observed baseline harm alongside high conditional susceptibility; they do not establish natural rare-event rates or demonstrate an autonomous cascade. The account motivates complementary defenses: strengthen resistance and recovery alongside prevention of spontaneous deviations, and audit and restrict unintended communication paths that can turn local failures into collective loss of control.