Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents
Seonghyeon Cho and Chanjun Park at Korea University show that the standard way of measuring whether agent skills help is confounded by selection bias, and introduce a matched-execution estimator that flips the conclusion for several models.
Ask this paper
The confound, stated plainly. Aggregate metrics compare retrieved against non-retrieved tasks, which are not the same tasks. Any difference mixes the effect of retrieval with the effect of which tasks trigger retrieval.
The estimator. Retrieval-Invoked Actual-Use Effect computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned only on tasks where the agent actually retrieved a skill.
The paradox it exposes. Across 17 LLMs on coding and math, models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, several models that look system-wide beneficial actively harm themselves on exactly the tasks where retrieval fired.
What this means for skill libraries. A skills directory that improves your aggregate score may be hurting you on every task it touches, and only a matched comparison can tell you which.
Abstract
Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.