Spurious Tool Use: When RL Agents Learn the Wrong Reason to Act
Yiwei Yang, Haoxiang Zhang, Pan Lu, Bill Howe and colleagues (UW, UCSD, Stanford) show that RL-trained agents learn to call tools because of superficial prompt cues, and fix it with a per-call necessity reward.
Ask this paper
Setup: Synthetic QA and math environments inject cues correlated with search or code tools during training but irrelevant to whether the tool is needed; Qwen2.5-7B-Instruct is trained with RL.
Shortcut rate: On counterfactual tests where the cue appears but the tool is unnecessary, spurious tool calls increase by up to 39%.
When it happens: Shortcuts form only after the agent already uses the target tool reliably, are amplified when cue meaning matches the tool, and largely disappear below 80% cue-task correlation.
Fix: A dense reward where an LLM judge rates the necessity of each tool call removes cue-driven calls without lowering task performance.
Abstract
Large language model (LLM) agents increasingly interleave natural language reasoning with external tools such as web search and code execution. These tool-use policies are often optimized via reinforcement learning (RL), which can amplify spurious correlations in the training data. In this work, we study when and why RL-trained agents learn shortcut tool-selection policies: invoking tools based on superficial prompt cues rather than genuine task requirements. We construct controlled synthetic environments combining factual question answering and mathematical reasoning tasks, and inject cues that are strongly correlated with specific tools during training but causally irrelevant to tool necessity. Across counterfactual evaluations where cues are present but the associated tools are not required, agents exhibit substantial shortcut behavior, with spurious tool invocation rates increasing by up to 39 percent. However, shortcut formation is not universal: across the conditions we test, it arises only when the agent has already learned to use the target tool reliably, suggesting that task competence, rather than dataset imbalance alone, is a key factor in shortcut learning. A swapped-cue analysis further shows that semantic alignment between cues and tools substantially amplifies this effect. To mitigate these failures, we introduce a dense, decision-level reward in which an LLM judge evaluates the necessity of each tool call. This tool-necessity reward effectively suppresses cue-driven tool use while preserving task performance, providing a practical approach to improving the robustness of LLM agent tool-use policies.