Agentic Reward Modeling

Reward agent "REWARDAGENT" - The authors introduce a modular system combining (1) a router to detect what checks are needed (factual accuracy, adherence to instructions, etc.), (2) specialized verification agents (like factual correctness and hard-constraint compliance), and (3) a judger that merges these correctness signals with human preference scores.

Factual checks via pairwise verification - Instead of verifying every claim in isolation, their system compares two candidate responses, identifies differing factual statements, and queries evidence (from the LLM's own parametric knowledge or a search engine). This process cuts costs while improving factual precision.

Constraint-following agent - To ensure instructions are followed (like response length or formatting), the system auto-generates and executes Python "checker" scripts. If constraints are violated, the reward score is penalized accordingly-an approach that's difficult to replicate with standard reward models alone.

Benchmarks & real-world gains - REWARDAGENT outperforms existing reward models on challenging tasks (RM-Bench, JudgeBench, plus a newly created IFBench for constraint compliance). Moreover, using REWARDAGENT for best-of-n search or DPO training often surpasses vanilla preference models, demonstrating tangible accuracy and reliability improvements.