How Data Scales in Agentic Reinforcement Learning: Laws and Synthesis Strategies
Bowei He, Yankai Chen, Xue Liu and colleagues (MBZUAI, McGill, CityUHK; NeurIPS 2026) fit scaling laws for agentic RL data along three axes, tasks, environments and trajectories, and derive a compute-optimal synthesis recipe.
Ask this paper
Axis ordering. Across model sizes, families, domains (math+code, SWE, web, UI/tool) and both PPO and GRPO, scaling exponents order as environment > task > trajectory.
What drives environment value. Splitting environment diversity into world diversity and structure diversity shows the advantage comes mainly from world diversity. Structure diversity is about as useful as task diversity.
Pricing synthesis methods. The fitted laws give each synthesis method a per-axis discount factor and a synthetic-to-real ratio at which collapse begins. These ratios differ by an order of magnitude across axes.
Recipe. A Chinchilla-style budget allocation validated at held-out scale gives up to 1.7x compute efficiency over the common practice of scaling trajectories.
Abstract
Pretraining scaling laws treat training data as a one-dimensional quantity: a token count. Agentic reinforcement learning (RL) inherits this scalar abstraction, but modern synthesis pipelines now generate \textit{tasks}, \textit{environments}, and \textit{trajectories} independently and at very different costs, turning data into a structured design space. This raises a question pretraining never had to answer: given a synthesis budget, \textit{which axis} should be scaled? We address this question end-to-end with three tightly coupled contributions. First, we decompose agentic data into three cardinality axes (task, environment, trajectory), plus a reward-density control variable, and fit per-axis scaling relations on different large language models (LLMs) across math+code, SWE, web, and UI/tool application domains. Across all model sizes, model families, domains, and RL algorithms (PPO and GRPO) we measured, the scaling exponents order as environment $>$ task $>$ trajectory. When environment diversity is decomposed into world and environment-structure diversity, its advantage is mainly driven by the former, while the latter is just comparable to the task diversity. Second, we use the fitted laws as a common ruler to price synthesis methods, capturing each method's efficiency as a per-axis discount factor and identifying critical synthetic-to-real ratios at which collapse begins; these ratios differ by an order of magnitude across axes. Third, we cast budget allocation as a constrained optimization under our laws and discounts, deriving a Chinchilla-style compute-optimal recipe and validating it at held-out scale. The empirical results demonstrate that following the recipe yields up to $1.7\times$ compute efficiency over the dominant ``scale-the-trajectories'' practice.