DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration
Yingda Shen, Zhizheng Wu and colleagues (The Chinese University of Hong Kong, Shenzhen; Amphion Technology) present DuplexAgent, a full-duplex voice system whose delegation harness is split into six editable modules, and Duplex-Harness-RSI, a loop that repairs those modules from simulated conversation traces.
Ask this paper
Six modules. Policy, Intake, Hold, Worker, Deliver and Memory control how a live voice model hands tasks to background reasoning and coding agents, and how updates, cancellations and results return to the user.
Simulator. A simulator generates timed test conversations with interruptions and mid-task changes, runs the system, and attributes each failure to a module.
Improvement loop. An Exam Planner chooses the next tests from attributed failures and a repair archive, a Harness Editor edits only the named modules, and paired checks on a fixed core and a protection set decide what is kept. The same delegation pool that serves users runs the loop.
Results. The ten-round checkpoint scores 85.3, 30.7 points above the initial harness and 23.4 above repeated editing without diagnosis or archive. Missed delegation falls from 30.3% to 1.3% of fixed-core items, and ignored cancellations fall by 81%.
Weights unchanged. All gains in spoken knowledge, tool use and collaboration come from harness edits, with model weights fixed.
Abstract
Voice agents are converging on a collaboration pattern: a full-duplex interaction model stays on the live channel as the entry to the conversation, while search, reasoning, and coding are handled through asynchronous delegation. A duplex model supports continuous listening and speaking, but complex reasoning and tool use may exceed its capabilities. A coding agent can plan and execute extended tasks, but its sequential interface is a poor fit for live conversation. Combining them requires a harness that coordinates task acceptance, progress, cancellation, replacement, and result delivery while keeping the conversation responsive. Existing harnesses often rely on coupled heuristics, making them difficult to improve systematically from evidence. We present DuplexAgent, a full-duplex collaboration system whose harness expresses this workflow as six editable modules, and Duplex-Harness-RSI, a closed loop that revises them from interaction traces. A simulator automatically generates timed test conversations, runs the system, and produces failure traces that identify the collaboration modules requiring repair. Reasoning LLMs and coding agents in the delegation pool also serve the improvement loop: the Exam Planner selects the next tests from observed weaknesses and the repair archive, and the Harness Editor proposes targeted module changes. The capabilities that serve the user thus also improve the system's coordination. Experiments on intelligence, agentic, and duplex benchmarks show that DuplexAgent combines continuous interaction with difficult reasoning and complex task execution, achieving stronger spoken-knowledge and executable-tool scores than the compared delegated systems while maintaining strong interruption response. A harness ablation further shows that this modular, verifiable loop outperforms the initial harness and repeated editing that lacks its diagnosis and repair archive.