DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression
DeepSeek-AI releases DeepSeek-V4.1-Flash, a 552B-parameter multimodal MoE built around a Causal Encoder-Decoder architecture that activates 16B parameters per decode token and 8B per prefill token, and cuts the resident KV cache to 890 bytes per token.
Ask this paper
Two KV compression mechanisms stack. Compressed Sparse Attention 2 reuses KV and index state across layers, and the main KV cache is stored in FP4. Together they bring the global HBM-resident footprint to 890 bytes per token, about one quarter of DeepSeek-V4-Flash.
SWA Bounded Replay handles the persistent tier. The cache that lives on SSD or in host memory drops to roughly one eighth of the V4-Flash footprint, which is the part that determines storage cost and transfer bandwidth for long-running agent sessions.
Prefill is where the asymmetry pays. Agent workloads are input-heavy, so activating 8B parameters during prefill instead of 16B lowers the dominant cost term directly rather than lowering decode cost that agents use less of.
Trained on 45T multimodal tokens. Post-training targets agentic use; the report puts it on par with closed frontier models on Terminal-Bench 2.1, DeepSWE v1.1 and AutomationBench, while noting a remaining gap to larger models on the hardest reasoning sets.
Checkpoints are public. Weights are on Hugging Face, so the architecture claims about cross-layer reuse and FP4 caching can be measured directly rather than taken from the report.
Abstract
The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth. Together, these compute, storage, and bandwidth demands constitute the primary bottleneck to further lowering deployment costs. To address this challenge, we introduce DeepSeek-V4.1-Flash, a multimodal Mixture-of-Experts (MoE) model with 552B backbone parameters and support for contexts of up to one million tokens. With its Causal Encoder-Decoder (CED) architecture, the model activates 16B parameters per token during decode but only 8B parameters during prefill, substantially improving cost efficiency for agentic workloads. To push the limits of KV cache compression, DeepSeek-V4.1-Flash combines cross-layer KV cache reuse in Compressed Sparse Attention 2 (CSA2) with FP4 KV caching. These designs reduce its global KV cache footprint (always in HBM) to 890 bytes per token, roughly 1/4 of the corresponding footprint of DeepSeek-V4-Flash. Further, through a dedicated deployment optimization known as SWA Bounded Replay, DeepSeek-V4.1-Flash reduces its persistent KV cache footprint (always on SSD or in host memory) to roughly 1/8 of that of DeepSeek-V4-Flash. Despite its much smaller KV cache footprint, the model delivers substantially better performance than the baseline. In addition, we streamline the DeepSeek-V4 architecture and introduce several efficient architectural extensions. We pretrain DeepSeek-V4.1-Flash on a multimodal corpus comprising 45T tokens and conduct comprehensive post-training, yielding strong performance across diverse text-based and multimodal agentic scenarios. Model checkpoints are available at https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash.