CURIO: Curiosity-Driven Test-Time Learning for Open-Ended Discovery
Tao Feng, Jiaxuan You (UIUC), Fangxu Yu (UMD), Pan Lu (Stanford) and colleagues introduce CURIO, a test-time RL method for open-ended discovery that adds a curiosity bonus so the policy keeps exploring directions that score low early.
Ask this paper
Problem. Test-time RL that strongly favors high-reward attempts can drop low-reward directions before their value shows, which limits open-ended discovery.
Curiosity model. An Intrinsic Curiosity World Model learns transitions in the policy's hidden-state space and gives prediction-error bonuses to sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight control how much the bonus affects updates.
Results. With Qwen3 backbones from 8B to 235B, three-run means beat a matched task-only RL control on five math objectives, match the best reported Circle Packing result, and improve single-cell denoising on both held-out corpora at every scale.
Size of gains. Relative gains reach 18.3% on Hadamard at 235B and 10.8% on denoising Score at 8B. The improvement is not uniform; ThetaEvolve scores higher on 8B Tabula.
Diversity. Generated programs show more structural variation than the task-only RL baseline at every stage.
Abstract
Open-ended discovery requires learning from repeated attempts while continuing to explore directions whose value is not yet apparent. Search with a frozen large language model (LLM) can reuse previous solutions in context, but cannot update the model from its successes and failures on the test problem. Reinforcement learning (RL) enables such adaptation; however, strongly favoring high-reward trajectories may suppress low-reward yet potentially promising directions too early. We introduce CURIO, a curiosity-driven test-time learning framework that complements task feedback with an Intrinsic Curiosity World Model (ICWM). The ICWM learns transitions in the policy's hidden-state representation and supplies prediction-error bonuses at sampled tokens outside the policy's top-k choices. Epoch normalization and an annealed weight regulate their contribution to the policy update. On six mathematical discovery tasks and single-cell denoising with Qwen3 backbones from 8B to 235B, three-run means improve over a matched task-only RL control on five mathematical objectives, match the best reported performance on Circle Packing, and improve denoising Score and mean squared error (MSE) on both held-out corpora at every tested scale. Relative gains reach 18.3% on Hadamard and 10.8% on denoising Score. Code-diversity measurements show greater structural variation among generated programs, supporting curiosity as a complementary exploration signal for learning in open-ended discovery.