Dream-RSI: Recursive Self-Improvement through Evolving Worlds
Tong Zheng and colleagues at the University of Maryland and Google DeepMind introduce Dream-RSI, which improves a coding agent's exploration policy by replaying its own past discovery trees as a simulator.
Ask this paper
Dilemma: Fixed exploration strategies stop working as search spaces grow, and online policy optimization needs expensive, delayed feedback over long rollouts.
Replay simulator: Accumulated discovery history is used as a simulator of the search space that has already been explored, giving cheap off-policy feedback on candidate exploration policies.
Loop: A light orchestration layer makes exploration explicit and programmable without changing the coding agent. Improved policies are deployed online, and new discoveries expand the simulator pool.
Results: Across algorithm engineering, mathematical optimization and GPU kernel engineering, Dream-RSI matches or improves discovery quality and reduces discovery cost in several settings.
Abstract
Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.