Bridging Offline and Online Reinforcement Learning for LLMs

Online Beats Offline – The study confirms that online and semi-online RL methods, such as Direct Preference Optimization (DPO) and Group Reward Policy Optimization (GRPO), significantly outperform their offline counterparts on both verifiable (math) and non-verifiable (instruction following) tasks. This highlights the importance of training on responses generated by an up-to-date model.

Semi-Online DPO Shines – Surprisingly, semi-online DPO, which periodically syncs the model weights, performs comparably to fully online DPO and GRPO. This suggests that pure online training may not be strictly necessary, and that less frequent synchronization can offer a good balance between performance and computational efficiency.

DPO and GRPO are Neck and Neck – When trained in an online setting, DPO and GRPO show very similar performance and convergence. This is a noteworthy finding, as DPO is often considered an offline method, while GRPO is designed for online learning.

Multi-Tasking for the Win – The paper demonstrates that jointly training on both verifiable and non-verifiable tasks leads to improved performance across the board, compared to training on either task alone. This suggests that combining different reward signals can lead to more robust and generalizable models.