MemoryTax: The Bitter Lesson of Agent Memory
Zedong Peng (Shanghai Jiao Tong University) runs a small, fully costed study asking whether purpose-built agent memory such as Mem0 and Zep still helps once a strong model can search raw history with its own tools, as coding agents do.
Ask this paper
Setup. One LoCoMo conversation (419 turns, 152 questions); four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5. One run per cell, and the author says so.
Memory helps only weak models. Mem0 beats full context only with GPT-4o-mini, and building its memory alone costs more than an entire full-context rollout.
Writer sets the ceiling. A stronger reader cannot recover information a weaker memory writer left out.
Agentic search wins. With Opus 5.5, the Pi coding agent tops the cost-accuracy frontier at 99% for under half the cost of full context, with its single miss due to a wrong answer key. Part of the lead comes from photo captions only agents could read; on the 88 photo-free questions, Pi and full context differ by one answer.
Abstract
Agent memory systems such as Mem0 and Zep decide what to keep before any question arrives; coding agents keep the raw history and let the model search it with its own tools. We ask whether designed memory still helps once the model is strong enough to search for itself. On one LoCoMo conversation (419 turns, 152 questions), we run four memory methods (full context, BM25, Mem0, Zep) and three coding agents (Pi, Codex, Claude Code), each with GPT-4o-mini, GPT-6 Luna and Claude Opus 5.5, and record what every rollout costs. This is a short empirical study, with one conversation and one run per cell. Agent memory is a costly patch for a weak model: Mem0 beats full context only with GPT-4o-mini, and its build alone costs more than a whole full-context rollout. Agent memory also has a ceiling, set by the model that writes it: a stronger reader does not recover what a weaker writer left out. With Claude Opus 5.5, Pi tops the cost-accuracy frontier, scoring 99% for less than half the cost of full context; its one miss is a wrong answer key. Part of that lead comes from photo captions that only the agents could read: on the 88 questions whose evidence has no photo, Pi and full context differ by one answer.