Self-Evolving Visual Agent via Margin Reward
Wenxi Yang, Yuzhong Zhao, Qixiang Ye and colleagues (UCAS, Microsoft Research Asia, Beihang) present Marvo, a challenger-solver loop that teaches a visual agent tool use from unlabeled images.
Ask this paper
Challenger and solver. A challenger writes question-answer pairs from unlabeled images, and a solver learns to answer them with tools through RL.
Margin reward. The challenger is rewarded by the difference in solver accuracy with and without tools, so it produces questions that need tools but are still solvable.
Results. Across agentic reasoning, visual search, math and general real-world reasoning benchmarks, Marvo improves average accuracy by up to 7.1%.
Stability. Across both model families, 5 of 42 per-round changes are negative under Marvo against 9 of 42 under R-Zero, and the largest single-round drop is 0.7 points against 5.5.
Abstract
Visual agents that interleave reasoning with tool invocation have posed a promising direction for multimodal tasks. However, learning effective tool invocation usually relies on curated question-answer pairs or expert demonstrations at scale, which are expensive to acquire. In this study, we propose Marvo, a simple-yet-effective self-evolution approach that enables learning tool invocation from unlabeled images. Marvo consists of two agents: (1) a challenger that generates question-answer pairs from unlabeled images, and (2) a solver that learns tool invocation by solving questions through reinforcement learning. To encourage effective tool invocation, a good training question should be difficult enough to warrant tool use, yet remain within the solver's capabilities. To this end, we introduce a margin reward to the challenger, defined as solver's performance difference with and without access to tools. With the margin reward, we create an adaptive curriculum that encourages the solver to learn effective tool invocation. Across benchmarks covering agentic reasoning, visual search, mathematical, and general real-world reasoning, Marvo improves average accuracy by up to 7.1\% --- demonstrating margin reward as a preferred way to scale question complexity without human supervision.