HySparse2: Hybrid Sparse Attention with Two-Level KV Sharing
Jianyu Wei, Yizhao Gao, Shijie Cao, Fuli Luo and colleagues on the LLM-Core team at Xiaomi present HySparse2, an attention architecture for agent workloads with long tool observations and short actions, which shares KV caches at two levels so that prefill can stop halfway through the network.
Ask this paper
KV Bridging. A YOCO-style split into a self-decoder (hybrid sliding-window attention) and a cross-decoder (hybrid sparse attention), where only the full-attention layers are bridged. The cross-decoder's full-attention KV caches are built from the self-decoder's full-attention hidden states, so prefill can exit after the self-decoder.
KV Reuse refinements. Sparse layers select individual tokens instead of 64-token blocks (1,024 global tokens), which on its own adds 6.57 points on RULER-v2, 8.14 on two-needle MRCR-v2 and 5.55 on GraphWalks at the same attention budget. A forced window of 128 recent tokens replaces the separate SWA branch, so local and global tokens share one KV cache.
Controlled comparison. HySparse2, HySparse and the Hybrid SWA design used in MiMo-V2 are trained as 80B-A3B MoE models on the same ~500B pretraining tokens and ~100B post-training tokens. After post-training, HySparse2 raises mean MRCR-v2 and RULER-v2 by 11.30 and 19.81 points over HySparse and has lower AgentPPL and LongPPL than both baselines at every length up to 256k.
Inference cost. At 1M tokens, prefill FLOPs fall 2.92x relative to HySparse and 5.02x relative to Hybrid SWA, with a smaller KV cache helped by MQA in place of GQA. HySparse and HySparse2 use five full-attention layers against nine for Hybrid SWA.
Trade-offs. The forced window costs 5.08 points on GSM8K and 4.99 on MRCR-v2 against a gated SWA branch, which the authors accept because it removes the local KV cache that would block early-exit prefill. At 290B-A8B, KV Bridging keeps quality broadly comparable, though DROP drops from 71.37 to 68.17.
Abstract
Long-horizon and multi-turn agents typically generate short actions and process long observations from tools and environments. This growing context demands efficient prefill, compact KV-cache storage, and accurate long-context retrieval. To meet these demands, we introduce HySparse2, a hybrid sparse attention architecture with two-level KV sharing. At the outer level, KV Bridging adopts a YOCO-style self-decoder and cross-decoder structure, but bridges only full-attention layers. The self-decoder uses hybrid sliding-window attention (SWA), while the cross-decoder uses hybrid sparse attention. The KV caches for full-attention layers in the cross-decoder are generated from the hidden states of full-attention layers in the self-decoder. At the inner level, HySparse2 retains HySparse's core KV Reuse design with two refinements. First, it replaces block-level sparsity with token-level sparsity for finer long-context retrieval. Second, it removes the separate SWA branch from sparse layers and instead forces a sliding window of recent tokens into the sparse selection. This two-level KV sharing allows all cross-decoder KV caches to be constructed from self-decoder hidden states. Prefill can therefore exit after the self-decoder, skipping all cross-decoder layers. On an 80B-A3B MoE model, HySparse2 outperforms HySparse and Hybrid SWA on long-context retrieval and multi-turn agentic tasks, while substantially reducing prefill computation and KV-cache storage.