Building py-kvcache: A Performance Characterization of External KV Caching for vLLM with NVMe SSDs
Joseph Kanichai, Tiziano De Matteis (Vrije Universiteit Amsterdam) and Animesh Trivedi (IBM Research) measure when loading KV cache from CPU or NVMe is faster than recomputing it in vLLM, and build py-kvcache, an offload connector that starts disk reads while requests are still queued.
Ask this paper
Tradeoff: For short prefixes or fast GPUs, recomputation can be faster than loading. Cache performance depends on transfer granularity, intermediate memory use and when transfers enter the request schedule, as well as on device bandwidth.
Design: py-kvcache uses asynchronous direct I/O, bounded shared staging memory and scheduler-aware preloading that overlaps disk reads with compute.
Results: At 80K tokens, loading from disk is 2.0x faster than LMCache, and preloading contributes 1.34x of that. With GPU, CPU and disk tiers enabled it is 1.23x faster than LMCache and within about 4% of native vLLM KV Offload.
Deployment advice: On Bailian trace replays an H100 keeps enough prefixes in GPU memory that the average request falls below the break-even point, so external KV caching should be decided for each setup.
Abstract
Prefix caching can reduce the time to first token (TTFT) of long-context LLM requests by reusing previously computed key-value (KV) states, but for short prefixes or fast GPUs, recomputation can be faster than loading from an external cache. We characterize this tradeoff in vLLM across GPU, CPU, and NVMe tiers using synthetic workloads, long-context benchmarks, production traces, and find that cache performance depends on transfer granularity, intermediate memory use, and when transfers enter the request schedule, not only on device bandwidth. These findings motivate py-kvcache, a vLLM KV Offload connector with asynchronous direct I/O, bounded shared staging, and scheduler-aware preloading, which starts disk reads while requests are still waiting, overlapping with compute. At 80k tokens, py-kvcache loading from disk is 2.0x faster than LMCache, with preloading contributing 1.34x. With GPU, CPU, and disk caching enabled, it is 1.23x faster than LMCache and within approximately 4% of the native vLLM KV Offload implementation. LongBench and SCBench show that these benefits extend to irregular prefix chains and multi-turn workloads. Bailian trace replays improve TTFT on a weaker GPU, but on an H100 the average request falls below the break-even point and GPU memory alone retains enough prefixes. External KV caching should therefore be treated as a setup specific admission decision. The py-kvcacheimplementation is available at: https://github.com/atlarge-research/py-kvcache.