AI Research Preference Models
Foster, Weston, Foerster and colleagues train preference models that rank an AI research agent's candidate experiments before any of them run, so a fixed GPU budget goes to the candidates most likely to pay off.
Ask this paper
The bottleneck is selection, not generation: An AIRA agent can propose far more candidates than it can execute, so progress depends on how it allocates a fixed execution budget rather than on how many ideas it produces.
RPMs predict promise without paying for it: Preference models built from frozen pretrained LLMs rank candidate solutions ahead of execution, avoiding evaluations that can take days of GPU time each.
Two variants: An inference-only model reasons over candidate plans, code and previously executed solutions; an agentic model additionally runs small-scale pilot experiments before committing budget.
Measured gains on AIRS-Bench: Integrated into AIRA-dojo, the two variants lift average normalized score from 0.684 to 0.711 and 0.729, with new state of the art on two tasks.
The budget result is the practical one: Both variants match the unguided agent's 24-hour performance in roughly 15 hours using under two-thirds of its execution budget.
Abstract
AI research agents (AIRA) can now carry machine learning experiments from proposal through implementation and evaluation. Yet progress on frontier tasks is throttled by the cost of evaluations that can consume days of GPU time. When an agent can propose far more candidates than it can afford to run, progress depends on its research preference: how it allocates a fixed execution budget across many candidates. We introduce AI Research Preference Models (RPMs) that predict which candidate solution is most promising, without paying the cost of running them all. We build RPMs from frozen pretrained language models in two variants: an inference-only model that reasons over candidate plans, code, and previously executed solutions, and an agentic model that additionally runs small-scale pilot experiments. Integrated into the AIRA-dojo research agent and evaluated on the machine learning research benchmark AIRS-Bench, the two variants increase the average normalized score from 0.684 to 0.711 and 0.729, respectively. Both reach the unguided agent's 24-hour performance in roughly 15 hours, using less than two-thirds of its execution budget, and together yield new state-of-the-art results on two AIRS-Bench tasks.