Language Models Learn to Improve Language Models
Minhua Lin, Suhang Wang, Hanqing Lu and colleagues (Penn State, UC Santa Cruz, Emory, UC Santa Barbara, Amazon) propose LM2LM, which trains an LLM with RL to propose language-model improvements, using a frozen research world model as a cheap surrogate reward and periodically grounding it with real runs.
Ask this paper
Cost problem. RL over LM-improvement proposals would need tens of thousands of training runs. At ten GPU-hours each, reward collection alone would cost hundreds of thousands of GPU-hours.
Research world model. A frozen LLM predicts a proposal's gain from real experimental records supplied in context and serves as the reward model.
Keeping the surrogate honest. Policy optimization drifts toward proposals the surrogate overestimates, so the method periodically executes top-ranked current-policy proposals and adds the measured outcomes to the world model's context.
Results. Across eight autoregressive and diffusion LM development environments, with real-execution evaluation on held-out tasks, LM2LM lets Qwen3.8-27B outperform GPT-6-sol even when GPT-6-sol gets 300 real experimental records in context, at 33% lower data-generation cost.
Transfer. The learned proposal ability generalizes to a model family not seen during training.
Abstract
Large language models (LLMs) are increasingly used to automate AI research, including language-model (LM) development, offering a potential path toward recursive self-improvement. However, turning experimental feedback into a learned ability to propose effective improvements across models faces two challenges. First, training such an ability with real feedback is expensive: reinforcement learning (RL) requires evaluating tens of thousands of proposals, each potentially requiring a full model-training run. Even at ten GPU-hours per run, reward collection alone would consume hundreds of thousands of GPU-hours, motivating a cheaper surrogate. Second, surrogate feedback can become unreliable as policy optimization favors overestimated proposals when the proposal distribution shifts. To address these challenges, we propose LM2LM, which trains LLMs with RL to generate effective LM improvement proposals. To provide scalable feedback, we use a research world model (RWM), a frozen LLM that predicts proposal gains from real experimental records provided in context, as the reward model for RL. To maintain feedback reliability, we periodically execute current-policy proposals ranked highly by the RWM and add their measured outcomes to its context. Across eight autoregressive and diffusion LM development environments, real-execution evaluations on held-out tasks show that LM2LM enables Qwen3.8-27B to outperform GPT-6-sol, even when GPT-6-sol is given 300 real experimental records in context, while requiring 33% lower data-generation cost. LM2LM also generalizes to a model family unseen during training.