Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Ming Zhong, Stacy Li, Nicholas Carlini and Matthew Jagielski (Anthropic Fellows Program, Anthropic) introduce Agent4Sci, a long-horizon testbed that asks frontier coding agents to improve the published best model on 20 scientific ML benchmarks across 11 domains, and then studies what kind of progress the agents actually make.
Ask this paper
Every benchmark improves. Frontier agents beat the published best on all 20 benchmarks. Claude Opus reaches a 10.4% mean relative improvement, and no run was flagged for academic misconduct.
The gains are engineering, not science. Trajectory analysis shows the improvements come almost entirely from standard ML-engineering tricks rather than domain insight. Three failure patterns recur: miscalibrated judgments of novelty, under-use of the available compute, and fast collapse into local hyperparameter and engineering sweeps.
Plug-in skills double the gain. Lightweight skills for research ideation, experiment management and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, and the resulting interventions are more varied and more domain-relevant.
Testbed, not leaderboard. The benchmark is framed as a way to track whether agents move from ML engineering toward domain insight; code and data are released on GitHub and Hugging Face.
Abstract
Frontier coding agents are increasingly framed as end-to-end automated researchers, and ML for science offers a demanding test of that promise: improving a scientific model often requires domain insight beyond routine ML engineering. We introduce Agent4Sci, a long-horizon testbed for measuring how far today's agents can improve the published best models on scientific ML benchmarks, spanning 20 benchmarks across 11 domains. Rather than ranking agents on a leaderboard, Agent4Sci asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short. Empirically, today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement and no run flagged for academic misconduct. These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight. Trajectory analysis surfaces three recurring limitations in this setting: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps. Lightweight plug-in skills for research ideation, experiment management, and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, while yielding more diverse, domain-relevant interventions. Agent4Sci gives the community a way to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code and data are available on GitHub (https://github.com/maszhongming/agent4sci) and Hugging Face (https://huggingface.co/datasets/MingZhong/agent4sci).