Multi-Agent Evolve
Free while signed in. Answers cite the passages they came from.
Multi-Agent Evolve (MAE) enables LLMs to self-improve their reasoning capabilities without human-annotated data through a co-evolving multi-agent framework. Three interacting agents (Proposer, Solver, Judge) instantiated from a single LLM undergo reinforcement learning optimization together, creating a scalable self-improving system that extends beyond game-based environments to general reasoning domains. - **Data-efficient self-improvement**: Addresses the critical limitation of existing self-play RL methods by eliminating dependence on human-annotated datasets. The co-evolving framework allows models to bootstrap their own reasoning improvements through internal agent interactions, making the approach practical for domains where labeled data is scarce or expensive. - **Three-agent architecture**: The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both outputs. This triangular interaction creates diverse training signals as each agent's improvement drives the others to adapt, establishing a dynamic self-reinforcing learning loop that continuously raises the difficulty and quality of training examples. - **General reasoning capability**: Unlike prior self-play approaches limited to game environments with clear win/loss signals, MAE operates across mathematics, reasoning, and knowledge Q&A tasks. This generalization demonstrates that co-evolution can work in open-ended domains without explicit reward structures. - **Proven efficiency gains**: Testing on Qwen2.5-3B-Instruct showed an average 4.54% improvement across multiple benchmarks. These results validate that the co-evolving dynamics genuinely enhance model capabilities rather than merely optimizing for specific evaluation metrics. - **Scalability without supervision**: The framework presents a path toward continuous model improvement with minimal human intervention. This addresses a fundamental bottleneck in applying RL to language models—the need for extensive human feedback or carefully curated reward signals for each new capability domain.
Data-efficient self-improvement: Addresses the critical limitation of existing self-play RL methods by eliminating dependence on human-annotated datasets. The co-evolving framework allows models to bootstrap their own reasoning improvements through internal agent interactions, making the approach practical for domains where labeled data is scarce or expensive.
Three-agent architecture: The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both outputs. This triangular interaction creates diverse training signals as each agent’s improvement drives the others to adapt, establishing a dynamic self-reinforcing learning loop that continuously raises the difficulty and quality of training examples.
General reasoning capability: Unlike prior self-play approaches limited to game environments with clear win/loss signals, MAE operates across mathematics, reasoning, and knowledge Q&A tasks. This generalization demonstrates that co-evolution can work in open-ended domains without explicit reward structures.
Proven efficiency gains: Testing on Qwen2.5-3B-Instruct showed an average 4.54% improvement across multiple benchmarks. These results validate that the co-evolving dynamics genuinely enhance model capabilities rather than merely optimizing for specific evaluation metrics.
Scalability without supervision: The framework presents a path toward continuous model improvement with minimal human intervention. This addresses a fundamental bottleneck in applying RL to language models—the need for extensive human feedback or carefully curated reward signals for each new capability domain.
Abstract
Reinforcement Learning (RL) has demonstrated significant potential in enhancing the reasoning capabilities of large language models (LLMs). However, the success of RL for LLMs heavily relies on human-curated datasets and verifiable rewards, which limit their scalability and generality. Recent Self-Play RL methods, inspired by the success of the paradigm in games and Go, aim to enhance LLM reasoning capabilities without human-annotated data. However, their methods primarily depend on a grounded environment for feedback (e.g., a Python interpreter or a game engine); extending them to general domains remains challenging. To address these challenges, we propose Multi-Agent Evolve (MAE), a framework that enables LLMs to self-evolve in solving diverse tasks, including mathematics, reasoning, and general knowledge Q&A. The core design of MAE is based on a triplet of interacting agents (Proposer, Solver, Judge) that are instantiated from a single LLM, and applies reinforcement learning to optimize their behaviors. The Proposer generates questions, the Solver attempts solutions, and the Judge evaluates both while co-evolving. Experiments on Qwen2.5-3B-Instruct demonstrate that MAE achieves an average improvement of 4.54% on multiple benchmarks. These results highlight MAE as a scalable, data-efficient method for enhancing the general reasoning abilities of LLMs with minimal reliance on human-curated supervision.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack