Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents
Xiaofang Yang and colleagues at Shanghai AI Lab argue that pre-install vetting cannot secure skill-augmented agents, because a malicious skill only acts once a concrete user task makes the unsafe action look useful, and implement the runtime guard itself as an installable skill.
Ask this paper
Why the threat is durable. Skills load as persistent runtime context, so a malicious one has a standing channel to steer future actions, leaking secrets, corrupting code, bypassing approvals, or staging data for later exfiltration.
Defense-as-Skill. SkillSonar runs alongside untrusted task skills and routes each sensitive action to allow, replan, or confirm by checking it against the user's task boundary, without modifying the agent runtime.
A dataset built for the setting. SCOPE-R covers 6 risk families and 21 sub-categories with 206 attack-confirmed malicious instances and 43 benign tasks, conditioned on tasks rather than on isolated prompts.
MCTS over an on-disk skill file. Runtime guard-skill evolution improves the guard through Monte-Carlo Tree Search over rollout feedback, which is an unusual and appealing target: the artifact being optimized is a readable markdown skill.
Strong numbers across two harnesses. On repeated GLM-5 runs, in-distribution attack success rate drops from 0.482 to 0.104 and out-of-distribution from 0.606 to 0.115, with transfer across victim models, held-out risk families, and external benchmarks, evaluated in Claude Code and OpenClaw.
Abstract
Skill-augmented agents load reusable skills as persistent runtime context, improving task performance but also giving malicious skills a durable channel for steering future actions. Such skills may leak secrets, corrupt code, bypass approvals, or stage data for exfiltration only after a concrete user task and workspace state make the unsafe action appear useful. This makes pre-install vetting insufficient and calls for runtime, task-conditioned protection. We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill. Our guard, SkillSonar, runs alongside untrusted task skills and checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision without modifying the underlying agent runtime. To study this setting, we construct SCOPE-R, a task-conditioned dataset covering 6 risk families and 21 sub-categories, with 206 attack-confirmed malicious instances and 43 benign tasks. We then improve SkillSonar on the SCOPE-R training subset using runtime guard-skill evolution, a Monte-Carlo Tree Search procedure that evolves the on-disk guard skill from feedback on the rollouts. Across Claude Code and OpenClaw, the evolved guard substantially reduces attack success while maintaining a favorable safety-utility trade-off. On repeated GLM-5 runs, SkillSonar reduces ID ASR from 0.482 to 0.104 and OOD ASR from 0.606 to 0.115. Further analyses demonstrate transfer across victim models, held-out risk families, and external benchmarks, as well as retained protection against adaptive attackers. Ablations further show that explicit safety responsibility assignment and the skill-native representation are both important to the observed gains.