SGD-KV: Summarization Guided KV Cache Compression
Zeyu Liu and colleagues present SGD-KV, which identifies attention heads specialized in hierarchical aggregation using a chunk-summarization diagnostic and allocates KV cache budget to them instead of applying a uniform heuristic.
Ask this paper
Heads have functional roles: existing KV compression scores tokens and ignores that different heads do different jobs. SGD-KV first asks which heads aggregate information hierarchically, then budgets accordingly.
The diagnostic is the contribution: a chunk-summarization task produces a summarization score per head, and the score distribution drives budget allocation, making the method head-aware rather than token-aware.
75% memory reduction at 1M context: state-of-the-art results on Qwen2.5-7B-1M and Qwen3-32B across long-context benchmarks with contexts up to a million tokens.
Why it matters: pairs naturally with the Random Attention result from Salesforce in the same block. One says token-level selection signal is near-worthless; this one says the head-level allocation signal is where the remaining headroom is.
Abstract
Large language models (LLMs) face severe memory bottlenecks in long-context inference due to the linearly growing size of key-value (KV) caches. Existing KV cache compression techniques typically rely on simple heuristics, overlooking the distinct functional roles of different attention heads. We present SGD-KV (Summarization-Guided KV Cache Compression), a head-aware framework that leverages a novel chunk-summarization diagnostic task to systematically identify and prioritize attention heads specialized in hierarchical information aggregation. Experiments on Qwen2.5-7B-1M and Qwen3-32B across diverse long-context benchmarks demonstrate that SGD-KV achieves state-of-the-art performance with contexts up to 1M tokens, while reducing KV cache memory usage by up to 75%. Our findings show that strategically allocating the KV cache budget based on the summarization score distribution of attention heads yields a superior efficiency-accuracy trade-off for long-context inference.