FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience
Zixun Huang, Kishan Panaganti, Haitao Mi and Leowei Liang propose FlowBalance, which lets a reasoning model learn from its own dense self-guidance but calibrates every guidance signal against the verifier's group advantage so false confidence gets reversed rather than reinforced.
Ask this paper
The inner loop is fragile for two opposite reasons: terminal verifiers are reliable but sparse, while dense same-model guidance is plentiful but can amplify confident errors and collapse onto one solution mode.
Sign-aware calibration: guidance is kept on positive-advantage trajectories, reversed on negative-advantage ones, and switched off entirely when the rollout group shows no outcome preference.
Trajectory balance instead of an imitation loss: the energy reweights a reference policy and profiled trajectory balance fits it with one log-partition estimate per group, avoiding a separate token-level imitation term.
Four proved properties: within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control, and an exact correction against false-positive self-guidance on rejected responses.
Avoids the known failure modes: beats FlowRL on Qwen3-4B and 8B, trains faster and more stably, sidesteps OPSD's response-length collapse, and keeps higher correct-strategy diversity on an AIME24 diagnostic.
Abstract
A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.