Where Should a Document Live: Context, Representations, or Parameters?
Nathanaël Carraz Rakotonirina, Momchil Hardalov, Gonzalo Iglesias and Adrià de Gispert (Amazon AGI) compare KV-cache and parametric ways of giving a model a document, across five knowledge-intensive benchmarks at matched storage budgets.
Ask this paper
Oracle setting: Cartridges, a trained compressed KV prefix, is the most accurate injection method at nearly every storage budget and beats parametric adapters by about 10 points.
Compaction: KV Compaction matches Cartridges only at low compression and falls about 10 points behind parametric methods above 50x compression.
Multi-document retrieval: With retrieved documents, Cartridges is the only method that matches in-context learning, 29 points ahead of parametric methods and 15 ahead of Compaction.
Forgetting: Cartridges, full fine-tuning and large MLP adapters are the only methods with catastrophic forgetting, about 6% on control benchmarks and 13% on coding; composing per-document artifacts remains the main open problem.
Abstract
To answer questions outside of their pre-training data, large language models (LLMs) need access to new information, which can be presented in the context window as documents, encoded into the model's parameters, or injected as latent representations. However, each of these methods comes with different efficiency, cost, and performance trade-offs, with no single winner. We present a controlled comparison of representation-based (KV-cache based) and parametric (fine-tuning-based) adaptation methods on five knowledge-intensive benchmarks. We show that in the oracle setting, Cartridges (KV) are the most accurate injection method at nearly every storage budget, outperforming parametric methods by 10 points. Compaction (KV) matches Cartridges only at low compression rates, lagging behind the parametric methods by 10 points at rates higher than $50\times$. In the more realistic multi-document retrieval scenario, Cartridges are the only method that matches in-context learning (ICL), leading the parametric methods by 29 points and Compaction by 15 points. Nonetheless, Cartridges are also the only method, besides full fine-tuning and large MLP adapters, that suffers from catastrophic forgetting, i.e., a 6% performance degradation on control benchmarks, with 13% in coding.