Back to Basics: Revisiting REINFORCE in RLHF
Free while signed in. Answers cite the passages they came from.
Cohere researchers argue that PPO is overkill for RLHF and that a simpler REINFORCE-style estimator works better in practice.
PPO is overkill: Many PPO features (clipping, value networks, GAE) are shown to be unnecessary in an RLHF context where episodes are short and reward variance is bounded.
REINFORCE / RLOO beats PPO: A straightforward REINFORCE variant with leave-one-out baselines outperforms PPO on RLHF benchmarks at lower compute and simpler hyperparameter tuning.
Also beats offline alternatives: The same approach outperforms newer offline/semi-offline methods like DPO and RAFT, pushing back on the "online RL is dead for RLHF" narrative.
Implication: Low-cost online RL with well-designed baselines is a practical path forward for alignment, especially for smaller teams that couldn't afford full PPO infrastructure.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack