WASP-Mem: Let Agent Organize Procedural Memory. Human Spend Effort on Diagnosis
Peizhi Niu and Ge Shi (UIUC and Amazon), with Christos Faloutsos and colleagues, introduce WASP-Mem, which distills agent execution traces into interlinked wiki pages of procedural memory, and run a controlled study of what makes such memory work.
Ask this paper
Method. Execution traces and feedback are consolidated into persistent, revisable wiki pages that merge repeated successes and diagnosed failures into reusable actions and recovery strategies.
Results. On TheAgentCompany, WASP-Mem raises checkpoint pass rate by 4.6 to 6.1 points over the strongest baseline across three models, using 57% to 61% less storage. Free-form WASP-Mem beats no memory by 10.9 to 14.5 points with 92 to 255 KB stores.
Prescriptive structure hurts. Human-defined schemas and explicit deduplication rules lowered performance instead of helping.
Diagnosis helps. One round of human diagnosis of failed traces beats four rounds of self-judgment by 7.0 to 10.9 points.
Division of labor. Let the agent organize its memory, and spend human effort on diagnosing failures.
Abstract
LLM agents struggle to automate complex enterprise procedures because static, pre-built harness cannot capture proprietary policies, undocumented behaviors, or state-dependent recovery paths that surface only during execution. Procedural memory accrued from agent experience offers a training-free path to closing this gap -- yet unlike episodic records or semantic facts, it must consolidate repeated successes and diagnosed failures into reusable actions and recovery strategies. We introduce WASP-Mem (Wiki-based Agent-Synthesized Procedural Memory), a nimble but effective method that distills execution traces and feedback into persistent, interlinked wiki pages, enabling continual agent learning. On TheAgentCompany benchmark, WASP-Mem raises the checkpoint pass rate by 4.6-6.1% over the strongest baseline we compared across three models while requiring 57-61% less storage. We further present a systematic controlled study spanning consolidation method, feedback type, trace ingestion scope, and refinement rounds. Our results reveal that prescriptive knowledge organization such as human defined schemas and explicit deduplication rules hurts performance rather than closing the procedural-knowledge gap. In contrast, human diagnosis converts failed traces into corrective knowledge and drives sustained improvement: a single round of human diagnosis outperforms four rounds of self-judgment by 7.0-10.9%. These findings point to a simple division of labor: let agents organize procedural memory, and invest human effort where it matters most -- diagnosis.