Making Every Tool Call Count: Necessary Tool-Evidence Path Rewards for Agentic Vision-Language Models
Xingming Long and colleagues introduce NTEP, an annotation scheme that names the necessary external evidence and the tool calls that must produce it, and NTEP-R, a reward that pays the agent per tool call rather than only on the final answer.
Ask this paper
Outcome rewards under-supervise the middle: training on final-answer correctness alone leaves both evidence acquisition and evidence utilization unsupervised, producing redundant calls and unread observations.
Reward on both sides of the call: the agent is scored for aligning pre-call intent with a necessary evidence goal, and for having its post-call summary actually match the evidence returned.
Non-repeated-goal regularizer: explicitly penalizes calls that revisit an already-satisfied NTEP goal, which is the concrete fix for tool-call thrash.
8B model, seven benchmarks: NTEP-8B improves search-oriented accuracy and tool-use efficiency simultaneously under a unified three-tool framework of cropping, image search and text search.
Why it matters: most agent RL still rewards the destination. Rewarding the evidence path is the more transferable idea, and it applies well beyond vision.
Abstract
Modern vision-language models (VLMs) can directly answer many image-grounded questions, yet they often struggle with complex queries requiring fine-grained visual details or external knowledge. To acquire this missing evidence, agentic VLMs invoke tools such as image cropping, image search, and text search. However, existing training paradigms primarily evaluate tool-use based on final answer correctness, leaving evidence acquisition and utilization insufficiently supervised. This leads to two critical shortcomings: (i) models frequently issue redundant or off-target tool calls that fail to gather necessary evidence, and (ii) even when appropriate tools are called, models often fail to extract the necessary information from the resulting observations. To address these limitations, we introduce the NTEP (Necessary Tool-Evidence Path), a novel annotation scheme that explicitly specifies the essential external evidence and corresponding tool calls for each query. Building upon this, we propose NTEP-R (NTEP Reward), a supervision mechanism ensuring that each tool invocation strictly advances the reasoning process toward the final solution. Specifically, our approach rewards the agent for aligning its pre-call intent with a necessary evidence-seeking goal, and for ensuring the information summarized from the post-call observation aligns with the necessary evidence. Furthermore, we introduce a non-repeated-goal regularizer to penalize redundant calls that revisit satisfied NTEP goals. Extensive evaluations on seven image-grounded benchmarks demonstrate that our 8B-parameter instantiation, NTEP-8B, significantly improves both search-oriented accuracy and tool-use efficiency within a unified three-tool framework. These results highlight the critical value of fine-grained tool-evidence path supervision for training robust agentic VLMs.