Compaction, decay and reinforcement in a two-tier conversational memory: a paired evaluation against retrieval-augmented generation
Osamudiamen Edogun, Joshua Obigwe and Irfan Mehmood (independent and University of Bradford) compare compacting each conversation turn into atomic facts against plain RAG over the transcript on LoCoMo, with a paired design and exact significance tests.
Ask this paper
Design. Both arms answer all 1,985 LoCoMo questions with the same stores, generator and judge, so McNemar's exact test applies to the 512 questions where they disagree.
Result. On the four categories comparable work reports, compaction scores 71.4% against 64.6% (p = 5.5e-08) using 34% fewer prompt tokens.
Where it loses. Including the adversarial category, the lead falls to 2.0 points and is no longer significant, because RAG leads adversarial by 14.6.
Mechanism. Compaction keeps what a fact asserts and drops how it was said. It leads by 9.6 points when the reference answer has one or two words and trails by 3.1 past eight words. Four negative results are reported with power calculations.
Abstract
A conversational agent that remembers has to decide what to keep. Retrieval-augmented generation keeps the transcript and searches it; the alternative is to compact each turn into atomic facts and search those. We compare the two on LoCoMo, with both arms answering all 1,985 questions in one window, against the same stores, with the same generator and judge, and both predictions on one row, which licenses McNemar's exact test on the 512 questions where they disagree. On the four categories comparable work reports, compaction scores 71.4 per cent against 64.6, a lead of 6.8 points (p = 5.5e-08) at 34 per cent fewer prompt tokens. On all five, including the adversarial set comparable work drops, the lead falls to 2.0 and stops being significant, because the baseline leads adversarial by 14.6. That loss has a structure. Our lead falls as the reference answer demands more source wording, 9.6 points ahead at one or two reference words and 3.1 behind past eight, with token overlap moving the same way, so the judge is not the cause. Compaction keeps what a fact asserts and discards how it was said, so it wins where the answer is a value and loses where it is a phrasing. We also report four negative results with their power calculations.