SkillsBench
Free while signed in. Answers cite the passages they came from.
SkillsBench evaluates whether LLM agents can generate their own procedural knowledge across 86 tasks spanning 11 domains, with curated Skills and deterministic verifiers. Testing 7 agent-model configurations over 7,308 trajectories, the benchmark reveals a critical gap: agents benefit enormously from consuming procedural knowledge but cannot reliably author it themselves. - **Curated skills boost performance significantly:** Providing curated Skills raises the average pass rate by 16.2 percentage points, with effects varying dramatically by domain, from +4.5pp in Software Engineering to +51.9pp in Healthcare. This shows that skill quality and domain match matter more than having skills at all. - **Self-generated skills provide no benefit:** On average, models that generate their own procedural knowledge show no improvement over having no skills. This finding is critical for self-improving agent architectures that assume models can bootstrap their own capabilities. - **Focused beats comprehensive:** Skills with 2-3 focused modules outperform comprehensive documentation. This suggests that retrieval precision matters more than coverage when augmenting agents with procedural knowledge. - **Smaller models close the gap:** Smaller models augmented with well-curated skills can match the performance of larger models operating without skill augmentation. This has direct cost implications for production agent deployments.
Curated skills boost performance significantly: Providing curated Skills raises the average pass rate by 16.2 percentage points, with effects varying dramatically by domain, from +4.5pp in Software Engineering to +51.9pp in Healthcare. This shows that skill quality and domain match matter more than having skills at all.
Self-generated skills provide no benefit: On average, models that generate their own procedural knowledge show no improvement over having no skills. This finding is critical for self-improving agent architectures that assume models can bootstrap their own capabilities.
Focused beats comprehensive: Skills with 2-3 focused modules outperform comprehensive documentation. This suggests that retrieval precision matters more than coverage when augmenting agents with procedural knowledge.
Smaller models close the gap: Smaller models augmented with well-curated skills can match the performance of larger models operating without skill augmentation. This has direct cost implications for production agent deployments.
Abstract
Agent Skills are structured packages of procedural knowledge that augment LLM agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark of 86 tasks across 11 domains paired with curated Skills and deterministic verifiers. Each task is evaluated under three conditions: no Skills, curated Skills, and self-generated Skills. We test 7 agent-model configurations over 7,308 trajectories. Curated Skills raise average pass rate by 16.2 percentage points(pp), but effects vary widely by domain (+4.5pp for Software Engineering to +51.9pp for Healthcare) and 16 of 84 tasks show negative deltas. Self-generated Skills provide no benefit on average, showing that models cannot reliably author the procedural knowledge they benefit from consuming. Focused Skills with 2--3 modules outperform comprehensive documentation, and smaller models with Skills can match larger models without them.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack