RECOMP (Retrieval-Augmented LMs with Compressors)

Extractive compressor: Selects the most useful sentences from retrieved documents, retaining the most relevant signal at a fraction of token budget.

Abstractive compressor: Generates a summary synthesizing information from multiple retrieved documents, compressing redundancy across sources.

6% compression rate: Achieves compression rates as low as 6% with minimal performance loss on language modeling and open-domain QA.

Selective augmentation: The training scheme learns to emit empty summaries when retrieved docs are irrelevant - a built-in mechanism for gracefully handling noisy retrieval.