When Tools Get in the Way: The Effect of Unnecessary Tool Availability on LLM Answering
Saanvi Paturi and colleagues at Spark AI Research show that giving a model a related but unnecessary tool makes it stop answering questions it can answer from its own knowledge, even when it rarely calls the tool.
Ask this paper
Benchmark: The benchmark has 500 query pairs across 10 domains. Each pair has a query that needs the domain tool and a closed-domain query that does not, and every closed-domain query has a tool-unavailable control.
Size of the drop: Across six models, the pooled answer rate on closed-domain queries falls from 98.2% without the tool to 63.5% with it available. Gemini 2.5 Flash-Lite falls from 99.4% to 23.4% while calling the tool in only 7.8% of trials.
Prior tool calls: A preceding tool interaction recovers 410 of the 1,056 lost answers but causes 232 new losses, so the effect of conversation history differs by model.
Fix: A one-sentence scope-aware system instruction raises the tool-available answer rate by up to 45.6 percentage points, at some cost to tool use where the tool is needed.
Abstract
Large language models (LLMs) are increasingly deployed with external tools that extend what they can do beyond their own knowledge. Tools help on tasks that need external information, but their availability may also change how a model handles questions that do not need them. Prior work has mostly asked whether models select and use tools appropriately; whether an unnecessary tool changes the correctness of answers has received less attention. We ask whether making a related but unnecessary tool available affects a model's ability to answer from its own knowledge, and whether a preceding tool interaction changes this behaviour. We construct 500 query pairs across 10 knowledge domains. Each pair consists of a tool query, which needs the domain's tool, and a closed-domain query, which does not. Six LLMs are evaluated with the tool unavailable, available, and available after a prior tool call. Across 3,000 baseline trials the pooled answer rate is 98.2%. When an unnecessary tool is available it falls to 63.5%, with large differences between models. The decrease occurs even when the tool is rarely called, so it cannot be explained by unnecessary tool invocation alone. A one-sentence scope-aware system instruction recovers most of the lost answers.