SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning
Wei Yang, Shawn Li, Jesse Thomason, Xuezhe Ma, Yue Zhao and colleagues (University of Southern California, with George Washington University) present SynCo, which trains a task Synthesizer and a Reasoner together with multi-agent RL so that the generated training tasks track the Reasoner's current ability.
Ask this paper
Joint optimization. The Synthesizer and Reasoner are separate models updated from the same online rollouts. Correctness rewards train the Reasoner; task quality, answer reliability and outcome-grounded teachability reward the Synthesizer.
Results. Across eight math benchmarks SynCo averages 78.72%, against 75.85% with a frozen Synthesizer and 74.96% for ScaleQuest-Math, the strongest external synthetic-data method. AIME24 and AIME25 each rise 6.66 points over the frozen Synthesizer.
Where gains come from. The Learn subset of previously unsolved problems rises from 27.11% to 32.23%, while already-solved problems stay at about 98.5%.
Teachability predicts usefulness. Only 16.2% of 2,940 verified tasks give a non-zero group-relative gradient. Teachability correlates with that property at r=0.934, while novelty correlates at r=-0.041, and ranking by quality or novelty selects useful tasks at 12.8% and 12.0%, below the 16.2% base rate.
Drift. The share of gradient-bearing tasks falls from 31.3% to 9.5% over training as more generated tasks become uniformly solvable, which is the mismatch the co-training is designed to correct.
Abstract
Self-evolving LLM agents promise to improve autonomously through continual interaction and learning, reducing their dependence on manually curated supervision. Realizing this promise requires not only updating the agent, but also evolving its training experience as its capabilities change. However, most existing pipelines rely on static datasets or separately updated synthesis models, causing previously useful tasks to become trivial while overly difficult tasks remain uninformative. This growing mismatch between agent capability and training experience limits sustained self-improvement. To address this problem, we propose SynCo, an agentic data synthesis co-training framework for self-evolving LLMs based on multi-agent reinforcement learning. SynCo jointly optimizes two independently parameterized agents: a Synthesizer that constructs training tasks from the Reasoner's evolving capability state, and a Reasoner that learns from the resulting experience. Each synthesized task induces multiple Reasoner rollouts whose outcomes provide complementary rewards to both agents. Correctness feedback improves the Reasoner, while task quality, answer reliability, and outcome-grounded teachability guide the Synthesizer. Their updates are fed back into subsequent synthesis rounds, allowing the task-solving policy and its training distribution to evolve together. Extensive experiments across eight mathematical reasoning benchmarks demonstrate that SynCo substantially outperforms a broad range of existing synthetic-data methods and controlled baselines, achieving the strongest overall performance while deriving most of its gains from previously unsolved problems.