GrowPage: On-Demand KV Budgeting for Efficient LLM Reasoning Serving
Qiankun Ma and colleagues point out that every KV compression method fixes the per-request budget in advance and only decides what to keep, then make capacity itself a runtime resource that grows on demand.
Ask this paper
Demand varies twice over: across requests, which need different KV capacities, and within a single request, whose attention demand evolves as the reasoning output grows.
Dual-timescale query summaries: lightweight recent and long-term attention summaries whose relative working sets estimate where demand is heading.
Compress or grow at each boundary: at a capacity boundary GrowPage either compresses within the current allocation or acquires another physical page when broader demand appears.
Built on PagedAttention: integrating at the page level preserves continuous batching and prefix caching, which is what makes it deployable rather than a research prototype.
Why it matters: it reframes long-output reasoning serving from a per-request budgeting problem to a scheduling problem, which is the framing that scales in a shared cluster.
Abstract
Long-output reasoning has made the key--value (KV) cache a critical memory bottleneck for efficient LLM serving. Existing KV compression methods usually rely on a predefined per-request budget and adjust only which KV states are retained, leaving the total capacity fixed throughout decoding. However, reasoning workloads exhibit substantial demand variation: different requests require different KV capacities, and the attention demand of an individual request evolves during generation. We introduce \textbf{GrowPage}, an on-demand KV budgeting framework that treats KV capacity as a runtime resource. GrowPage maintains lightweight dual-timescale query summaries to capture recent and long-term attention behaviors, and uses their relative attention working sets to estimate demand evolution. At each capacity boundary, GrowPage either compresses KV states within the current allocation or acquires an additional physical page when broader demand emerges. By integrating with PagedAttention's page-level memory abstraction, GrowPage preserves continuous batching and prefix caching. Experiments on reasoning benchmarks across multiple models show that GrowPage achieves a superior performance--throughput trade-off over existing approaches.