Does Your Agent's Memory Survive a Model Upgrade? A Controlled Study of Memory Portability
Ankit Goyal and Jaideep Ray at LinkedIn run a controlled study of what happens to an agent's memory store when the model reading it changes, comparing verbatim long context, chunked RAG, model-written notes, and a fixed-schema knowledge graph.
Ask this paper
The setup. 48 synthetic histories with randomized answer codes, exact scoring, and two sub-10B open-weight models, so the writer and reader can be swapped independently.
Fixed schema transfers, compressed notes do not. KG-fixed accuracy changes by +0.0004 +/- 0.0020 after a writer swap, while compressed NOTES shift asymmetrically by +9.91 or -13.28 points depending on migration direction.
Mixed embedding indexes cost most of the gain. A 50/50 mixed index captures only 4.96 of the 11.90-point improvement that full re-embedding delivers.
The deficits have different causes. 80% of the NOTES deficit comes from information lost at construction time; 81% of the RAG deficit comes from retrieval failure.
Repair needs the source. Store-only repair of NOTES failed to reach 90% recovery in all 48 cases, while retaining the raw source history enabled recovery in 34 of 48 for one direction.
Abstract
Model upgrades are routine; memory migrations are not. An agent can keep the same memory store and still forget: a new model may interpret old notes differently, mixed embedding versions may break retrieval, and repair may fail without the original evidence. We compare memory as the same history is preserved verbatim for long-context reading (LC-RAW), divided into chunks for retrieval-augmented generation (RAG), compressed by a model into natural-language notes (NOTES), or normalized into a fixed-schema knowledge graph (KG-fixed). The study uses 48 synthetic histories with randomized answer codes, exact scoring, and two open-weight models with sub 10 billion parameters. Our measurements show that fixed-schema structures transfer reliably, with KG-fixed accuracy changing by only $+0.0004 \pm 0.0020$ following a writer swap. Conversely, compressed NOTES exhibit high model coupling, with accuracy shifting asymmetrically by $+9.91$ or $-13.28$ percentage points depending on the specific migration direction. In RAG systems, partial embedding migrations using a 50/50 mixed index capture only a 4.96-point accuracy improvement, forfeiting the majority of the 11.90-point gain achieved through full re-embedding. Diagnostic decomposition attributes 80% ($0.467 \pm 0.014$) of the NOTES accuracy deficit to information lost during initial construction, whereas retrieval failures drive 81% ($0.364 \pm 0.012$) of the RAG deficit. Finally, store-only repair of NOTES fails to reach a 90% performance recovery target in all 48 test cases, whereas retaining the raw source history enables successful recovery in 34 of 48 cases for one tested direction. These findings highlight the necessity of direction-specific migration testing, strict embedding space isolation, and the retention of source histories for memory repair.