LOCI: A Locator-Critic with Refinement Loop
Walid Bousselham, Mathilde Caron, Arsha Nagrani and Cordelia Schmid at Google DeepMind argue that VLM failures on hard visual tasks come from failing to locate the relevant detail, not from weak high-level reasoning, and fix it with a two-agent loop that needs no training.
Ask this paper
The diagnosis reframes the problem. A VLM that cannot find the critical region produces fluent reasoning on top of flawed perceptual grounding, which looks like a reasoning failure and is not one.
Visual search and evidence verification are separated. A Locator agent proposes candidate visual evidence and a distinct Critic agent judges its relevance and sufficiency, so neither role has to do both jobs.
The refinement loop is the mechanism. The two agents iterate, improving the evidence until the Critic judges it adequate to answer the question, which makes the process self-correcting rather than single-shot.
Gains hold on open and proprietary models. Qwen3-VL improves 12.1 on V*, 5.8 on HR-Bench and 11.2 on VisualProbe-Hard, and Gemini 2.5 Pro improves 8.9, 4.3 and 4.8 on the same three.
Training-free, so it is an inference-time harness change rather than a model change, and it applies to any VLM behind an API.
Abstract
Vision-Language Models (VLMs) still struggle on tasks requiring complex visual understanding. We argue that the core issue is not high-level reasoning, but instead failing to locate critical details in the image. Due to this shortcoming, VLMs generate often plausible but incorrect reasoning based on flawed perceptual grounding. To address this, we propose Locator-Critic (LOCI), a training-free framework that decouples visual search from evidence verification. LOCI employs a Locator agent to propose candidate visual evidence and a separate Critic agent to evaluate its relevance and sufficiency. These agents engage in an iterative refinement loop, progressively improving the evidence until it is adequate to answer the given question. This decoupled, self-correcting process yields substantial performance gains, achieving state-of-the-art results on multiple complex visual benchmarks. LOCI improves accuracy for both open-weight models like Qwen3-VL (+12.1 on V*, +5.8 on HR-Bench and +11.2 on VisualProbe-Hard) and proprietary models like Gemini 2.5 Pro (+8.9 on V*, +4.3 on HR-Bench, +4.8 on VisualProbe-Hard).