AutoRL: A Multi-Agent System towards Autonomous Reinforcement Learning

Researchers from Meituan, CUHK, ShanghaiTech and Fudan present AutoRL, a multi-agent system that supervises a running RL training job the way an RL researcher would, as a step toward recursive self-improvement.
Ask this paper
Roles. Perceptors watch training dynamics, rollouts, evaluations and efficiency, and trigger deliberation on anomalies or stagnation. Three Proposers diagnose and refine interventions with shared evidence. An Orchestrator approves changes and applies them by hot update or checkpoint resumption.
Math gains. Against a static GRPO recipe on Qwen3, AIME 2024 rises from 46.67% to 66.67%, AIME 2025 and HMMT 2025 each improve by 11.67 points, and IMO-AnswerBench by 9.75 points. Qwen3-1.7B improves AIME 2025 from 18.33% to 30.00%.
Agentic coding. On SWE-related tasks with LongCat-Flash-Lite, pass@1 rises from 52.29% to 65.13% across an unattended long RL run with interruptions.
Recipe repair. Given a deliberately flawed recipe (weak grader, no KL, 131K-token responses), AutoRL detects and fixes the problems. It beats a general-purpose agent on most tasks.
Abstract
Recursive self-improvement (RSI) is one of the most promising paths toward an AI singularity for superintelligence. Achieving this vision requires AI systems to undertake the full training pipeline for iterative improvement. Reinforcement learning (RL) plays an essential role in this cycle to improve model capabilities and generalization, yet automating RL remains difficult due to unstable training dynamics and the need for sustained supervision. We introduce AutoRL, a multi-agent system for automating RL training, as a cornerstone for future RSI. Different from conducting research by end-to-end complete experiments, AutoRL monitors the evolving training state and integrates process-level supervision and research into an ongoing RL training, following the practical workflow of an RL researcher. Specifically, AutoRL consists of three RL-specific agentic roles. Perceptors track training dynamics, rollouts, evaluation, and efficiency, triggering deliberation when they detect anomalies, degradation, or stagnation. Three complementary Proposers diagnose problems and refine candidate interventions through shared evidence and checks. The Orchestrator reviews their recommendation and applies approved changes through online hot updates or checkpoint-based resumption. Experiments on Qwen3 and LongCat-Flash-Lite demonstrate the superiority of AutoRL over static RL recipes and a general agent. The research records show that existing agents can handle unattended RL supervision and dynamic adjustment. The difficulty for future RSI lies in developing fundamental methods that are valuable for resource-consuming validation.