Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
Haoyue Liu, Xiaoyu Ma and Ye Chen (CUHK Shenzhen and FNii-Shenzhen) show that GRPO is structurally mismatched when the tool-subset action space is small enough to enumerate, and replace sampling with exact expectation over the full space.
Ask this paper
Where GRPO breaks: In specialist scientific domains a small set of recurring capabilities covers the field, so the space of tool subsets is combinatorial but enumerable, while GRPO still estimates the action expectation from a handful of sampled rollouts.
The failure worsens as training succeeds: As the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training.
Full-Group Policy Optimization: FGPO scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space rather than a sample of it.
Frozen-reasoner calls leave the training loop: The reward of each question-subset pair is precomputed into an exhaustive table, removing reward-model calls from training entirely.
Results: FGPO beats GRPO in all 15 settings across five frozen reasoners and three genomic benchmarks, by 6.75 points on average and up to 14.20, while an on-demand GRPO schedule would need 2.4 times as many frozen-reasoner evaluations. On GenomeQA it cuts invoked tools per question from 2.36 to 1.40.
Abstract
Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.