CLaRa
Free while signed in. Answers cite the passages they came from.
CLaRa introduces a unified framework for retrieval-augmented generation that performs embedding-based compression and joint optimization in a shared continuous space. The approach addresses key RAG limitations around long contexts and disjoint retrieval-generation optimization. - **SCP data synthesis:** Uses question-answer and paraphrase supervision to create semantically rich compressed vectors that remain retrievable for downstream tasks. - **End-to-end optimization:** Trains reranker and generator simultaneously using a single language modeling objective. Gradients flow through both modules via a differentiable top-k estimator. - **Theoretical grounding:** The unified optimization approach theoretically connects retrieval relevance with answer quality, aligning what gets retrieved with what improves generation. - **Results:** Achieves state-of-the-art compression and reranking performance across multiple QA benchmarks, often surpassing text-based fine-tuned baselines.
SCP data synthesis: Uses question-answer and paraphrase supervision to create semantically rich compressed vectors that remain retrievable for downstream tasks.
End-to-end optimization: Trains the reranker and generator simultaneously using a single language modeling objective. Gradients flow through both modules via a differentiable top-k estimator.
Theoretical grounding: The unified optimization approach theoretically connects retrieval relevance with answer quality, aligning what gets retrieved with what improves generation.
Results: Achieves state-of-the-art compression and reranking performance across multiple QA benchmarks, often surpassing text-based fine-tuned baselines.
Abstract
Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge but still suffers from long contexts and disjoint retrieval-generation optimization. In this work, we propose CLaRa (Continuous Latent Reasoning), a unified framework that performs embedding-based compression and joint optimization in a shared continuous space. To obtain semantically rich and retrievable compressed vectors, thereby reducing the document length fed into the generator, we introduce SCP, a key-preserving data synthesis framework based on question answering and paraphrase supervision. CLaRa then trains the reranker and generator end-to-end via a single language modeling loss, with gradients flowing through both modules using a differentiable top-k estimator. Theoretically, this unified optimization aligns retrieval relevance with answer quality. Experiments across multiple QA benchmarks show that CLaRa achieves state-of-the-art compression and reranking performance, even at a text compression rate of 16, outperforming text-based fine-tuned baselines.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack