EvoRS: On-Policy Self-Evolution of Reward Systems for Open-Ended Reinforcement Learning
Weiyuan Li and colleagues (Fudan University) propose EvoRS, where an agentic designer rewrites the reward system during RL training from on-policy rollouts and reward traces, instead of keeping rubrics fixed.
Ask this paper
Reward as a program: The reward system is represented as an executable Reward-DAG, so the designer can change criteria, scoring mechanisms and how signals are combined.
On-policy updates: The designer inspects current rollouts and reward traces to detect hacking and loss of discriminability, then edits the DAG.
Result: On writing and roleplay EvoRS gives the best quality under all three judges, beating the policy by 2.107 and 4.767 points, with less reward hacking.
Ablation: A comprehensive but fixed reward system does not stay reliable on these open-ended tasks.
Abstract
Open-ended reinforcement learning often relies on rubric-based rewards for tasks without directly verifiable answers. Yet the policy and reward system form a dynamic feedback loop: as the policy optimizes the current reward, an initially useful reward system may become unreliable due to reward hacking or reduced response discriminability. The reward system should therefore evolve rather than remain fixed during training. Existing dynamic-rubric methods adapt evaluation criteria, but reward failures can also arise from scoring mechanisms or signal composition. We introduce EvoRS, a self-evolving RL framework that evolves the reward system from on-policy experience, representing it as an executable Reward-DAG. Specifically, an agentic designer updates this system from on-policy rollouts and reward traces to maintain train-time reliability. Across writing and roleplay, EvoRS achieves the best quality under all three judges, outperforming the policy by \(2.107\) and \(4.767\) points, respectively, while reducing reward hacking and coverage failures and preserving reward informativeness. Ablations confirm that a comprehensive fixed reward system cannot remain reliable in open-ended tasks and must evolve throughout training.