Aligning Autoresearch: Co-Improvement between Agents and Humans
Tianjian Li, Chuanyang Jin, Seungone Kim, Jason Weston and colleagues (FAIR at Meta, with Johns Hopkins and CMU) study co-improvement between research agents and humans, covering human direction of benchmark creation, a reward-hacking benchmark, training against human simulators, and rubric-based RL for text generation.
Ask this paper
HackBench. The benchmark holds 169 trajectories from real agent runs. A median trajectory has 264 commands and one hacked command, so monitors must find sparse evidence in over a million characters.
Monitor scores. Claude Opus 5 is the best hack localizer at 64.88 F1, followed by Muse Wonder at 61.47. Every other model stays below 53, and all monitors miss many hacks.
Human direction. On Rebuttal Bench, fine-grained human feedback lowers model scores on the created benchmark from 80 to 90 with no feedback to 39.1 to 51.2, producing harder benchmarks that pass validity checks.
AutoInteract. Training agents with human simulators raises Pass@1 from 6.0 to 20.6 and SSR from 1.8 to 16.5 in one reported setting, ahead of RL on the source tasks.
Abstract
Autoresearch agents can now attempt long-horizon AI research end-to-end, i.e., self-improvement. However, best results are still obtained by research agents interacting with human researchers, and improving AI's ability to perform this co-research will only reinforce this effect---a process termed co-improvement (Weston and Foerster, 2025). Further, aligning with human expectations is likely a safer path for research agents, especially given recent agent reward hacking phenomena (OpenAI, 2026b). In this work we address these issues in four ways: (i) we empirically study the role of humans in directing autoresearch agents, focusing on the domain of benchmark creation, and find humans still play a pivotal role particularly in the ideation phase; (ii) we also develop a benchmark, HackBench, that shows current autoresearch agents are not aligned and are unable to achieve hack localization sufficiently; (iii) we present a method to train agents for improved human interaction by using human simulators as a proxy during reinforcement learning (RL); and finally (iv) we develop a method to improve model text generation, and hence agent communication, by RL with rubrics aligned with human experts. Together, our training and evaluation recipes provide a path toward more capable and aligned (co-)research agents.