Scaling Agent RL via Experience Synthesis
Free while signed in. Answers cite the passages they came from.
Meta researchers introduce DreamGym, a unified framework that synthesizes diverse training experiences to enable scalable reinforcement learning for LLM agents without costly real-environment rollouts, addressing fundamental barriers of expensive interactions, limited task diversity, unreliable rewards, and infrastructure complexity. - **Reasoning-based experience model**: Distills environment dynamics into textual state space predicting transitions through chain-of-thought reasoning, enabling scalable rollout collection without pixel-perfect simulation. - **Experience replay with co-evolution**: Integrates offline demonstrations with online synthetic interactions, retrieving top-k similar trajectories to reduce hallucinations while staying aligned with evolving agent policy. - **Curriculum-based task generation**: Adaptively generates challenging variations using reward-entropy heuristic to identify feasible yet difficult tasks maximizing information gain without manual verification. - **Dramatic non-RL-ready performance**: On WebArena, DreamGym outperforms all baselines by 30%+ across three backbones, providing the only viable RL approach where traditional methods fail. - **Matches traditional RL with zero real interactions**: Purely synthetic training matches GRPO/PPO performance versus 80K real transitions. Sim-to-real transfer achieves 40%+ gains using <10% real data. - **Sample efficiency and guarantees**: Training time reduced to 1/3-1/5 of real-environment RL. Theoretical analysis shows gap depends on reward accuracy and domain consistency, not strict state reconstruction.
Reasoning-based experience model: Distills environment dynamics into a textual state space, predicting transitions through chain-of-thought reasoning, enabling scalable rollout collection without pixel-perfect simulation.
Experience replay with co-evolution: Integrates offline demonstrations with online synthetic interactions, retrieving top-k similar trajectories to reduce hallucinations while staying aligned with evolving agent policy.
Curriculum-based task generation: Adaptively generates challenging variations using a reward-entropy heuristic to identify feasible yet difficult tasks, maximizing information gain without manual verification.
Dramatic non-RL-ready performance: On WebArena, DreamGym outperforms all baselines by 30%+ across three backbones, providing the only viable RL approach where traditional methods fail.
Matches traditional RL with zero real interactions: Purely synthetic training matches GRPO/PPO performance versus 80K real transitions. Sim-to-real transfer achieves 40%+ gains using <10% real data.
Sample efficiency and guarantees: Training time reduced to 1/3-1/5 of real-environment RL. Theoretical analysis shows the gap depends on reward accuracy and domain consistency, not strict state reconstruction.
Abstract
While reinforcement learning (RL) can empower autonomous agents by enabling self-improvement through interaction, its practical adoption remains challenging due to costly rollouts, limited task diversity, unreliable reward signals, and infrastructure complexity, all of which obstruct the collection of scalable experience data. To address these challenges, we introduce DreamGym, the first unified framework designed to synthesize diverse experiences with scalability in mind to enable effective online RL training for autonomous agents. Rather than relying on expensive real-environment rollouts, DreamGym distills environment dynamics into a reasoning-based experience model that derives consistent state transitions and feedback signals through step-by-step reasoning, enabling scalable agent rollout collection for RL. To improve the stability and quality of transitions, DreamGym leverages an experience replay buffer initialized with offline real-world data and continuously enriched with fresh interactions to actively support agent training. To improve knowledge acquisition, DreamGym adaptively generates new tasks that challenge the current agent policy, enabling more effective online curriculum learning. Experiments across diverse environments and agent backbones demonstrate that DreamGym substantially improves RL training, both in fully synthetic settings and in sim-to-real transfer scenarios. On non-RL-ready tasks like WebArena, DreamGym outperforms all baselines by over 30%. And in RL-ready but costly settings, it matches GRPO and PPO performance using only synthetic interactions. When transferring a policy trained purely on synthetic experiences to real-environment RL, DreamGym yields significant additional performance gains while requiring far fewer real-world interactions, providing a scalable warm-start strategy for general-purpose RL.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack