Fork Where the Model Changes Its Mind: Belief-Shift Branching for Tree-Structured Reinforcement Learning
Bin Lei and colleagues at Salesforce AI Research and the University of Minnesota place the forks of tree-structured RLVR rollouts at the step where the model's answer belief shifts most, which gives step-level credit where the outcome is still undecided.
Ask this paper
Why fork placement matters: Each fork costs extra samples, so budgets allow only a few per chain. A fork placed after the outcome is settled produces siblings that agree and carry almost no credit signal.
Belief-shift branching: The method reads the model's answer belief at candidate step boundaries and forks just before the largest change between consecutive beliefs, targeting the pivots of the value curve.
Three instantiations: A black-box probe, a logit-lens depth profile and a learned activation direction cover different access levels. None needs step-level supervision, and the probe costs about 1% of step compute on math and under 5% on code.
Validation: Against Monte-Carlo value curves, a belief-shift signal ranks first in all eight model-benchmark panels, ahead of entropy, structural and LLM-judge placement.
RL gains: Across three model families it leads every math aggregate, with +2.6 aggregate and +2.9 on AIME 2026 for OLMo-3-7B over the strongest baseline, and +6.5 on LiveCodeBench-medium.
Abstract
Tree-structured rollouts give critic-free reinforcement learning with verifiable rewards (RLVR) step-level credit: fork a chain at an intermediate point, and sibling outcome differences estimate step value. Each fork adds sampling cost, so realistic budgets typically allow only a few forks per chain. A fork placed where the outcome is already largely settled yields siblings that mostly agree and provide almost no credit signal; hence, for a given tree size, where forks are placed largely determines how much step-level RL can gain. Most existing mainstream methods place forks by structure, such as fixed lengths, midpoints, and delimiters, or by next-token entropy. We formalize fork placement as locating the \emph{pivots} of the chain's value curve, where the expected outcome turns. We propose \emph{belief-shift branching}: read the model's answer belief at candidate boundaries and fork just before the step where consecutive beliefs diverge most. Three instantiations, none needing step-level supervision, span access levels: a black-box probe, a logit-lens depth profile, and a learned activation direction, which is fit offline and therefore used only in the validation before RL training. The signal only \emph{places} forks, and the probe costs about $1\%$ of step compute on mathematics and under $5\%$ on code when it runs inside the rollout engine. In that validation, against Monte-Carlo value curves, a belief-shift signal ranks first in each of the eight model$\times$benchmark panels, ahead of entropy, structural, and LLM-judge baselines. In RL across three model families and two domains, belief-shift forking leads every mathematics aggregate, on OLMo-3-7B by $+2.6$ aggregate and $+2.9$ on AIME 2026 over the strongest baseline, and sweeps every OLMo code column, by $+6.5$ on LiveCodeBench-medium.