Cliff: Learning Process Rewards from the First Mistake
Peixuan Han, Runhui Wang and colleagues at AWS propose Cliff, a reward shaping method that asks an off-the-shelf teacher LLM to find only the first mistake in a rollout, then converts that single index into dense token-level advantages.
Ask this paper
One judgment instead of a per-step score: The observation is that once reasoning first goes wrong, grading everything after it adds little, because the suffix is conditioned on an invalid prefix. So you only need the cliff edge.
No reward model, no matched reasoning style: Unlike process reward modeling this needs no specialized PRM, and unlike on-policy distillation it does not assume teacher and student reason the same way.
Positive prefix, negative suffix: The rollout splits into a correct prefix that gets positive advantage and an incorrect suffix that gets negative feedback, at token granularity.
Works with weak teachers: Across 12 scenarios Cliff beats on-policy distillation by 15 percent and standard GRPO by 7 percent, holding up even when the teacher is of modest capability.
Why it matters: This is the cheapest credible route from sparse outcome reward to dense process supervision, and 'modest teachers suffice' is what makes it practical outside frontier labs.
Abstract
Reinforcement learning with verifiable rewards (RLVR) has emerged as a powerful paradigm for large language model (LLM) post-training, but its reliance on coarse outcome rewards leads to limited guidance on intermediate reasoning processes. Existing approaches such as process reward modeling and on-policy distillation introduce additional constraints, such as reliance on a specialized reward model or assuming identical reasoning patterns between teacher and student. Nevertheless, we observe that once a reasoning process first goes wrong, evaluating the subsequent reasoning provides limited additional information, as it is already conditioned on an invalid prefix. Therefore, we propose Cliff, a reward shaping strategy that utilizes an off-the-shelf LLM as a teacher to identify the first mistake in each rollout. As a result, the rollout is naturally decomposed into two parts: a correct prefix and an incorrect suffix. Cliff then converts this signal into token-level advantages, assigning positive advantages for the correct prefix and negative feedback afterward. Experiments across 12 different scenarios demonstrate that Cliff consistently improves reasoning performance, outperforming on-policy distillation by 15% and standard GRPO by 7%, even with teachers of modest capability. Furthermore, we analyse the role of ``ground truth'' in Cliff and investigate its training dynamics. These results establish Cliff as a simple, general and effective approach for improving RLVR with richer, fine-grained supervision.