Augmenting LLMs with Long-term Memory (LongMem)

Memory-augmented training: Dedicated adaptation training teaches the LLM to retrieve and use its memory of long past context.

ICL over long history: Enables in-context learning that spans far longer contexts than the model's raw attention window.

Decoupled retrieval: Separates the retrieval mechanism from the main model, allowing memory to grow without increasing model size.

Long-context direction: Part of 2023's multi-pronged attack on context-window limits, complementary to position interpolation and ring attention.