Training AI Co-Scientists Using Rubric Rewards
Free while signed in. Answers cite the passages they came from.
This paper from Meta Superintelligence Labs presents a scalable method to train language models to generate better research plans without expensive human supervision or real-world execution. The approach automatically extracts research goals and goal-specific grading rubrics from scientific papers, then uses reinforcement learning with self-grading to improve plan generation. - **Automated data extraction:** Research goals and grading rubrics are automatically extracted from papers across ML, medical, and arXiv domains. Human experts validated that 84% of rubric items capture necessary requirements for good research plans. - **Self-grading with privileged information:** A frozen copy of the initial model acts as a grader, using extracted rubrics as privileged information to evaluate plans. This creates a generator-verifier gap that enables training without external supervision. - **Strong human validation:** In a 225-hour study with ML experts, the finetuned Qwen3-30B model's plans were preferred over the initial model for 70% of research goals, with experts rating them as sounder and more likely to lead to better outcomes. - **Cross-domain generalization:** Models trained on one domain generalize significantly to others. The medical-finetuned model achieved 15% relative improvement on ML tasks and 17.5% on arXiv tasks, suggesting the approach learns generally desirable research plan qualities. - **Competitive with frontier models:** The finetuned 30B model becomes competitive with Grok-4-Thinking, achieving 12-22% relative improvements across domains, though GPT-5-Thinking remains the top performer.
Automated data extraction: Research goals and grading rubrics are automatically extracted from papers across ML, medical, and arXiv domains. Human experts validated that 84% of rubric items capture necessary requirements for good research plans.
Self-grading with privileged information: A frozen copy of the initial model acts as a grader, using extracted rubrics as privileged information to evaluate plans. This creates a generator-verifier gap that enables training without external supervision.
Strong human validation: In a 225-hour study with ML experts, the finetuned Qwen3-30B model’s plans were preferred over the initial model for 70% of research goals, with experts rating them as sounder and more likely to lead to better outcomes.
Cross-domain generalization: Models trained on one domain generalize significantly to others. The medical-finetuned model achieved 15% relative improvement on ML tasks and 17.5% on arXiv tasks, suggesting the approach learns generally desirable research plan qualities.
Competitive with frontier models: The finetuned 30B model becomes competitive with Grok-4-Thinking, achieving 12-22% relative improvements across domains, though GPT-5-Thinking remains the top performer.
Abstract
AI co-scientists are emerging as a tool to assist human researchers in achieving their research goals. A crucial feature of these AI co-scientists is the ability to generate a research plan given a set of aims and constraints. The plan may be used by researchers for brainstorming, or may even be implemented after further refinement. However, language models currently struggle to generate research plans that follow all constraints and implicit requirements. In this work, we study how to leverage the vast corpus of existing research papers to train language models that generate better research plans. We build a scalable, diverse training corpus by automatically extracting research goals and goal-specific grading rubrics from papers across several domains. We then train models for research plan generation via reinforcement learning with self-grading. A frozen copy of the initial policy acts as the grader during training, with the rubrics creating a generator-verifier gap that enables improvements without external human supervision. To validate this approach, we conduct a study with human experts for machine learning research goals, spanning 225 hours. The experts prefer plans generated by our finetuned Qwen3-30B-A3B model over the initial model for 70% of research goals, and approve 84% of the automatically extracted goal-specific grading rubrics. To assess generality, we also extend our approach to research goals from medical papers, and new arXiv preprints, evaluating with a jury of frontier models. Our finetuning yields 12-22% relative improvements and significant cross-domain generalization, proving effective even in problem settings like medical research where execution feedback is infeasible. Together, these findings demonstrate the potential of a scalable, automated training recipe as a step towards improving general AI co-scientists.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack