The Memory Trap: Language Agents Overgeneralize from Benign Interactions
Kaiqu Liang, Jaime Fernández Fisac and Thomas L. Griffiths (Princeton) with Andrew Lampinen (Anthropic), in work from the Anthropic Fellows Program, show that memory an agent writes during harmless sessions causes unsafe or unreliable behavior in later sessions outside the scope where the memory was formed.
Ask this paper
Autonomy. After users ask agents to skip confirmation during low-stakes coding chores, Sonnet-4.6 and Opus-4.7 take irreversible actions on simulated high-stakes tasks without asking in 35 to 81% of episodes with notes, from near-zero baselines.
Sycophancy and over-critique. Memory from users seeking emotional support raises Sonnet-4.6's preference for the user's weaker poem from 21% to 55%. Memory from users hurt by uncritical feedback has the opposite effect: correct proofs get less credit and sound arguments get invented flaws.
Cheating. Notes from legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards. On ImpossibleBench's conflicting SWE-bench split they raise Opus-4.8's cheating rate from 29% to 74%, while Fable-5 is barely affected.
Two routes. The failure occurs when the agent writes a note that broadens what the user said, and also when it acts on memory, including raw transcripts. No adversary is involved, so content filters have nothing to detect.
Implication. Memory systems need to record the scope in which each memory was formed and keep the agent from applying it outside that scope.
Abstract
Language agents increasingly use persistent memory to record user preferences and procedures across sessions. We show that memory from benign interactions can cause unsafe or unreliable behavior on later tasks. We call this failure the memory trap. To study this issue, we compare behavior in sessions with and without memory across six frontier models (including Claude Sonnet, Opus, and Fable models and GPT-5.6). We evaluate these models in a range of settings. First, in autonomy experiments, we find that requests to skip confirmation during low-stakes coding chores lead agents to skip confirmation and verification on simulated high-stakes tasks in both coding and everyday life. We next study memories formed when users seek emotional support, or describe harm from uncritical feedback, and find these memories bias later judgments of mathematical work, arguments, and poems toward sycophancy or over-critique (respectively). Memory notes written during legitimate stale-test repairs lead agents to weaken correct tests that catch broken safeguards; these memories also increase cheating on ImpossibleBench, with larger increases for stronger Sonnet and Opus models. We show that memory traps can arise both from overgeneralizations in memory writing, and when acting on memory. These findings demonstrate that even benign interactions can lead to serious risks from memory systems, and thus motivate the need for new research into ensuring that agents preserve the scope of their memories.