PaperGym: Rubric-Centered Evolution for Research-Plan Generation
Yuhan Wang and colleagues at Zhejiang University, with Kaitao Song at Apple, turn each scientific paper into a full RL environment by synthesizing the question from goal and background while deriving the criteria from method and experiments, cutting criterion leakage to 3.7%.
Ask this paper
Leakage is the flaw being fixed: Existing rubric pipelines draw question and criteria from the same content, so the reward is earnable by paraphrase. Splitting the sources drops leakage from 11.90 to 34.10% down to 3.7%.
The rubric is used twice: First as privileged context for a self-teacher via OPSD, then as the reward for GRPO. The ordering matters: the paper shows the reverse ordering and either stage alone both underperform.
Criteria span the right axes: Methodological innovation and experimental design, rather than a single scalar per rollout.
Consistent gains across three model sizes: Five-benchmark averages up 5.6, 5.0 and 4.8 points for Qwen3-1.7B, 4B and 8B.
Head-to-head against an existing corpus: With the recipe fixed, PaperGym-20k-trained models win 58.1% of three-way comparisons against 28.2% for RubricHub Science, and the trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6.
Abstract
Research planning is the decisive capability of AI scientists. Yet a research plan admits no verifiable answer, so reinforcement learning lacks the environment it requires: tasks paired with a critic. Rubrics extracted from scientific papers can supply the critic. Existing pipelines, however, draw the question and the criteria from the same content, so the reward can be earned by paraphrase. The rubric is further compressed into a single scalar per rollout. We introduce PaperGym, a unified framework that turns each research paper into a complete training environment. PaperGym exploits the structure of a paper: the question is synthesized from the research goal and background, while the criteria are derived from the method and experiments. The criteria span methodological innovation and experimental design, and criterion leakage falls to 3.7%, versus 11.90% to 34.10% in existing datasets. Training uses the rubric twice: first as privileged context for OPSD's self-teacher, then as the reward for GRPO. Across Qwen3-1.7B/4B/8B, this schedule outperforms supervised fine-tuning, either stage alone, and the reverse ordering, improving five-benchmark averages by +5.6, +5.0, and +4.8 points. With the recipe held fixed, models trained on PaperGym-20k win 58.1% of three-way comparisons, against 28.2% for RubricHub Science. The trained Qwen3-8B reaches 73.48 on ResearchQA, above the far larger Kimi K2.6. We release the pipeline, the 20,000-instance corpus PaperGym-20k, and the benchmarks PaperGym-Innov and PaperGym-Design.