Long-range Language Modeling with Self-Retrieval

End-to-end retrieval training: Unlike retro-fitted RAG, trains the retriever and LM jointly from scratch for long-range consistency.

Long-form coherence: Targets tasks requiring retrieval of distant past context within a long document, not just factual lookup.

Architecture innovation: Introduces training procedures and architectural choices that make joint training stable and efficient.

Long-context RAG: Presaged the research direction of treating RAG and long-context as complementary rather than competing solutions.