Scientist World: Grow a Research World for AI-Scientists
Taiming Lu, Sachin Konan and Zhuang Liu (Princeton) build Scientist World, in which each research paper becomes an executable environment where an AI-Scientist is scored against the method the paper's authors proposed.
Ask this paper
Pipeline. A multi-agent pipeline with no human in the loop reproduces each paper's method until it recovers the reported result, defines a research task that cannot be solved by shortcuts, packages it to run on any machine, and verifies it with independent agents.
Scoring. Frontier-Normalized Improvement (FNI) measures how much an AI-Scientist's method improves on the reproduced human method. The first release has 1,000 verified environments.
Results. AI-Scientists beat the paper's reproduced method in 84% of improve attempts and 74% of innovate attempts. Fable 5.1 and Fable 5 rank first and second at designing new methods, and GLM 5.2 is the closest open model.
Traces as training data. The world holds 6,400 hours of research traces. Training on them raises a small model's research-ideation score by 158%, to the level of frontier models.
Growth. New papers are added continuously, so the evaluation measures research on problems a model has not seen rather than recall.
Abstract
AI-Scientists arrived, but a world for them to grow in did not. Human scientists take theirs for granted $-$ centuries of open problems and shared ideas, waiting in the scientific literature. To AI-Scientists, that world is readable but not runnable, so we build Scientist World for them: every research paper becomes an executable environment. Our autonomous pipeline extracts each paper's open-ended research question and reproduces its proposed method as the baseline; an AI-Scientist entering the environment is evaluated against what the human authors achieved. We release the first 1,000 verified environments. With the world as an evaluation, we found that frontier AI-Scientists can already surpass the human researchers 74% of the time. As a record of how AI does science, it contains 6,400 hours of research traces, with every idea, experiment, failure, and success. As a training ground, we are able to raise (+158%) a small model to match frontier models in research ideation. At last, AI-Scientists' own world has arrived, grown from and growing with science itself.