Back to Basics: Revisiting REINFORCE in RLHF
Cohere researchers argue that PPO is overkill for RLHF and that a simpler REINFORCE-style estimator works better in practice.
Ask this paper
PPO is overkill: Many PPO features (clipping, value networks, GAE) are shown to be unnecessary in an RLHF context where episodes are short and reward variance is bounded.
REINFORCE / RLOO beats PPO: A straightforward REINFORCE variant with leave-one-out baselines outperforms PPO on RLHF benchmarks at lower compute and simpler hyperparameter tuning.
Also beats offline alternatives: The same approach outperforms newer offline/semi-offline methods like DPO and RAFT, pushing back on the "online RL is dead for RLHF" narrative.
Implication: Low-cost online RL with well-designed baselines is a practical path forward for alignment, especially for smaller teams that couldn't afford full PPO infrastructure.