KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpoints
Xi Shi and Qian Lou (University of Central Florida) build KVShareArena, a benchmark for reusing KV caches when the reused text is not a prompt prefix, which is the case for retrieval-augmented servers and for multi-agent coordinators reading reports written by other agents.
Ask this paper
The condition existing benchmarks assume: Serving systems reuse caches only when the reused text sits at the start of the prompt. A RAG server assembles a different chunk set per query and an agent coordinator reads other agents' reports, so the cache carries wrong positions and never attended to the other sources.
A single recovery metric: Every method is scored by the fraction of the gap it recovers between no cache and full recomputation, and charged compute, memory and per-request latency with the cache already in hand. The one-time cost of building a cache is reported separately.
Position correction is enough until multi-source: Correcting positions requires no recomputation and suffices for single-source questions. When a question needs several sources at once, only methods that re-encode part of the cache or that train recover half to two thirds of the gap, and unrepaired caches can score below using no cache at all.
Compression fails on agent reports: Cache-compression methods that are harmless on a single prompt fall significantly behind plain position correction on freshly written agent reports.
Cross-checkpoint result: When a different checkpoint of the same model family wrote the cache, training-free methods are barely affected while an adapter trained on one checkpoint's caches loses quality. Harness, frozen querysets and cost accounting ship as a pip package with a public leaderboard.
Abstract
LLM serving systems already reuse KV caches, but only when the reused text sits at the very start of the prompt. Two growing workloads break this condition: a retrieval-augmented generation server assembles a different set of retrieved chunks for every query, and a multi-agent coordinator reads reports written by other agents. Reused inside a new prompt, a cache carries the wrong positions and never attended to the other sources. The cache may also have been written by a different checkpoint of the same model family, which changes the stored values. Repair methods for such caches have appeared in three separate communities, each measured on its own terms, and existing benchmarks test only exact-prefix reuse, where nothing is lost. KVShareArena benchmarks KV-cache reuse across prompt contexts and model checkpoints on retrieved chunks and agent reports. It scores every method by the fraction of the gap it recovers between no cache and full recomputation, and charges compute, memory, and per-request latency with the cache in hand, reporting the one-time cost of building a cache separately. We find that correcting positions, which needs no recomputation, is enough until a question needs several sources at once. There, only methods that pay, by re-encoding part of the cache or by training, recover half to two thirds of the gap; unrepaired caches can be worse than no cache. Cache-compression methods that are harmless on a single prompt fall significantly behind position correction on freshly written agent reports. These patterns hold across three model boards. When a different checkpoint wrote the cache, training-free methods are barely affected, while an adapter trained on one checkpoint's caches loses quality. Harness, frozen querysets, and cost accounting ship as a pip package with an automated submission workflow and a public leaderboard.