Evolving in Thought Space: Training a Small Model at Test Time Unlocks Better Discoveries
Chonghe Jiang, Ao Qu, Jinhua Zhao, Paul Pu Liang and colleagues (MIT, with Stanford, ByteDance, NUS and others) propose Guidance-TTT, which trains only a small guidance model at test time to propose strategy changes while a larger frozen model writes the executable solutions.
Ask this paper
Split roles. A PUCT archive search picks a parent solution, the compact guidance model proposes a high-level modification, and the frozen executor implements it. Verifier reward updates only the guidance model through a group-relative RL objective.
Why split. TTT-Discover trains the solution-generating model itself, which is expensive at large scale and mixes credit for strategy and implementation in one reward. Training only the short strategic decisions separates the two.
Results. Without web access, Polyomino reaches 91.89, above four-agent web-enabled CORAL (89.40) and the human reference (89.10). Lasso improves over SimpleTES from 0.1243 to 0.1739, a 39.8% gain.
Cost. Adapting an 8B guidance model instead of the 120B executor gives comparable Polyomino scores (83.19 vs 83.72) at about 58% lower cost, roughly $120 against $283.
Abstract
Open-ended scientific discovery often requires repeatedly proposing and evaluating candidate solutions. LLM-based systems can support this process by generating and refining executable solutions from verifier feedback. Methods such as TTT-Discover use test-time training (TTT) to update the solution-generating LLM from verifier feedback, adapting its generation policy to improve subsequent proposals on the target problem. However, this becomes expensive when reliable execution requires a large model, since training must maintain gradients, optimizer states, and policy statistics while repeatedly generating long, structured outputs. It also complicates credit assignment: outcome-level verifier feedback must jointly evaluate the high-level strategy and its low-level implementation. In this work, we introduce Guidance-TTT, which separates these roles. A compact guidance model is trained at test time to propose high-level strategic changes, while a frozen execution model implements them as complete executable solutions. At each step, the system selects a promising previously discovered solution, proposes a change, executes and verifies it, and updates only the guidance model using an adaptive group-relative RL objective. This concentrates test-time learning on short strategic decisions while retaining the implementation capability of a substantially stronger model without adapting it. Without web access, Guidance-TTT produces strong solutions across four distinct domains: combinatorial optimization (Polyomino Packing), heuristic programming (AHC058), machine learning (Lasso), and GPU kernel optimization (TriMul). Across these tasks, it outperforms the best solutions reported in prior work while remaining competitive with state-of-the-art results on public online leaderboards. Code is available at https://github.com/Human-Agent-Society/reef/tree/guidance-ttt-support.