What Matters for Aggressive Decoding-Time KV Eviction? Temporal Aggregation and Ranking Preservation
Bo Zeng and colleagues show that the temporal rule aggregating KV scores across decode steps, usually treated as an implementation detail, dominates the scoring function everyone is publishing about.
Ask this paper
EMA aggregation makes scorer differences indistinguishable: under aggressive compression, exponential-moving-average aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level.
Which scorers actually differ: value-norm and entropy variants stay correlated with attention and retain nearly identical sets, whereas KeyDiff, key norm, recency and a learned scorer change the ranking and degrade substantially.
InertiaKV and InertiaKV-Lazy: an EMA-based decoding-time evictor plus a periodic-refresh variant that reaches 1.34 to 1.46x decode throughput over full refresh.
Score-Free decoding as a reference point: score the full context once at the first decode step, freeze the ranking, and average quality moves by +0.03 while all subsequent scoring disappears.
Careful about its own claim: the authors state this identifies temporal aggregation and ranking preservation as distinct design factors and does not imply scoring quality is irrelevant in general.
Abstract
Decoding-time KV cache compression research focuses heavily on designing better token scoring functions, while the temporal rule that aggregates scores across decode steps is often treated as an implementation detail. Under aggressive KV compression, we find that exponential-moving-average (EMA) aggregation makes approximately order-preserving scorer modifications largely indistinguishable at the eviction-set level. Value-norm and entropy variants remain highly correlated with attention and produce nearly unchanged retention sets, whereas KeyDiff, key norm, recency, and a learned scorer alter the ranking and degrade substantially. We associate this stability with the evaluated aggregation, which couples layer weighting and temporal retention. Building on this observation, we introduce InertiaKV, an EMA-based decoding-time eviction method, and InertiaKV-Lazy, its periodic-refresh variant, which yields 1.34-1.46x decode throughput relative to full refresh InertiaKV. We also study Score-Free decoding as a separate empirical operating point: it scores the full context once at the first decode step, freezes that ranking, and incurs an average quality change of +0.03 while removing all subsequent scoring. Across six open-weight backbones and the LongBench, LongBench-v2, and RULER benchmarks, the results identify temporal aggregation and ranking preservation as distinct, consequential design factors; they do not imply that scoring quality is irrelevant in general.