Edit, Don't Imitate: Minimal-Edit On-Policy Distillation for Multimodal Reasoning
Wen Ye, Ailin Deng, Hengduo Li and colleagues (Meta Superintelligence Labs, with USC, UC Berkeley and Cornell) train small vision-language models to stop misreading diagrams with minimal-edit corrective on-policy distillation, which uses reference figure descriptions only during training.
Ask this paper
Problem. In visual math, a VLM that misreads the diagram reasons correctly from false premises. Smaller models suffer most, and giving them an accurate textual description closes much of the gap to larger models, but that description is not available at test time.
Method. A teacher that sees the reference figure description and a verified solution scores incorrect student rollouts under a minimal-edit objective. Only the tokens the teacher disfavors contribute to the corrective loss, and correct rollouts receive task-reward RL.
Result. Across model scales the trained student, with no description at test time, meets or exceeds the base model that is given the reference description at inference. Gains are largest on low-level geometric perception and transfer to external math and perception benchmarks.
Teacher size. Because the teacher only evaluates corrections instead of writing its own solution, a frozen same-size teacher still works, although a larger teacher gives bigger gains on most metrics.
Abstract
Visual mathematical reasoning can fail before reasoning begins: when a vision-language model (VLM) misreads a diagram, subsequent reasoning proceeds from false premises. This perceptual bottleneck is especially pronounced in smaller models, while accurate textual descriptions can substantially narrow the gap to larger models. Because obtaining such descriptions at inference requires additional annotation or model calls, we propose minimal-edit corrective on-policy distillation. Using reference figure descriptions and verified reference solutions as training-only information, a teacher scores incorrect student rollouts under a minimal-edit objective, and only teacher-disfavored tokens contribute to the corrective loss. Correct rollouts receive task-reward reinforcement learning. This preserves on-policy supervision while avoiding full-response imitation. Across model scales, the method requires no descriptions at test time and meets or exceeds the corresponding base model supplied with a reference figure description at inference. It also improves perception-oriented performance, particularly low-level geometric perception, and transfers to external mathematics and perception benchmarks. Because the minimal-edit teacher evaluates corrections rather than generating its own solution, the method remains effective with a frozen same-size teacher, although its gains are smaller than with a larger teacher on most metrics.