Open Problems and Limitations of RLHF

Scope: Catalogs issues across the entire RLHF pipeline - preference data collection, reward modeling, policy optimization, and evaluation.

Fundamental limitations: Discusses issues that can't be solved by incremental engineering alone, including the difficulty of specifying human preferences completely.

Reward hacking taxonomy: Organizes the many varieties of reward hacking seen in practice, from sycophancy to specification gaming.

Research agenda: Argues for investment in alignment approaches beyond RLHF that can address its structural limitations - a precursor to DPO and related methods.