Environment Evolution for Terminal Agents
Zhiyuan Fan and colleagues on Tencent's Hunyuan team argue that co-evolving training environments from on-policy rollouts runs out of signal as the model improves, and propose evolving environment difficulty off-policy on a generation-by-generation schedule instead.
Ask this paper
The co-evolution trap: methods that synthesize environments from weaknesses exposed during rollouts inherit the policy's blind spots, which limits generalization and dries up learning signal precisely as the model gets strong.
Three evolution directions derived from the objective: the difficulty axes are taken from the multi-turn learning objective rather than picked heuristically, then implemented through a loop-engineered multi-agent harness.
Verified against frontier models: rollout experiments with Hy4 preview, Claude Opus 5 and GPT-5.6 Sol confirm the evolved environments are genuinely harder, which is the right way to validate a difficulty generator.
Large downstream gains: simple long-horizon RL on Qwen3.6-27B and Qwen3.6-35B-A3B improves Terminal-Bench 2.1 by 14.4 and 18.0 points respectively.
Why it matters: environment supply is becoming the binding constraint on agent RL. Making difficulty a schedulable, off-policy knob is the more scalable framing.
Abstract
Scaling interactive and verifiable environments is critical for training terminal agents. As frontier models become more capable, environments synthesized from scratch become less challenging and thus provide limited learning signals. Recent co-evolution methods iteratively synthesize environments near the model's learnable frontier based on weaknesses exposed during rollouts. However, their dependence on on-policy rollouts limits generalization and the continuous provision of learning signals as the model becomes stronger. In this paper, we propose environment evolution, which incrementally increases environment difficulty off-policy and schedules the evolved environments generation by generation during training to provide continuous learning signals. We derive three evolution directions that influence environment difficulty from the multi-turn learning objective and then implement evolution along these directions through a loop-engineered multi-agent harness. Quantitative rollout experiments with Hy4 preview, Claude Opus 5, and GPT-5.6 Sol show that environment evolution consistently produces more difficult environments. We validate its effectiveness on Qwen3.6-27B and Qwen3.6-35B-A3B through simple long-horizon RL training, improving their performance by 14.4 and 18.0 percentage points on Terminal-Bench 2.1, respectively.