Appreciation Teaches Unauthorized Execution: Self-Evolution Degrades the Robustness of Personal Agents to Indirect Prompt Injection
Xinyan Wang, Xiaogeng Liu, Jun Shao and Chaowei Xiao (University of Wisconsin-Madison and Johns Hopkins University) test whether learning from user feedback weakens personal agents against indirect prompt injection, using a controlled two-phase method on OpenClaw.
Ask this paper
Method. In adaptation, the user asks the agent to complete a task described in external content and gives feedback. In testing, a third party places a similar task in new content that the user did not authorize.
Skill evolution. Average attack success rises by 14.0 to 48.8 percentage points against the base agent.
RL. Average attack success rises by 3.0 to 39.3 percentage points.
Cause. The authors name the cause authorization conflation. The agent learns to execute tasks from external content without requiring user authorization.
Defenses. Delimiting tool outputs leaves substantial degradation. Under skill evolution, feedback that separates authorized from unauthorized execution prevents most of the increase.
Abstract
Self-evolving personal agents act for users and learn from user feedback. However, if an agent obeys an instruction that an untrusted third party hides in content, the third party gains control of the agent. Such indirect prompt injection (IPI) poses severe risks. We investigate whether self-evolution degrades the robustness of personal agents to IPI. To study this question, we design a controlled two-phase methodology. In the adaptation phase, the user asks the agent to read external content and complete the task described in it, a common request in daily use. The agent then learns from the user's feedback and improves itself as designed. In the test phase, a third party writes a similar task into new external content, but the user's test request does not authorize the injected task. Using this methodology, we build multiple tasks with diverse test instances in different domains for a real personal agent. Our extensive experiments reveal that self-evolution substantially degrades robustness to IPI. Specifically, compared with the base agent, which does not evolve from user feedback, the average attack success rate rises by 14.0--48.8 percentage points under skill evolution and by 3.0--39.3 percentage points under reinforcement learning, posing safety risks. The code is available at https://github.com/SaFo-Lab/Evolve-IPI