DeepSeek-V3.2-Exp
Free while signed in. Answers cite the passages they came from.
DeepSeek adds a fine-grained sparse attention mechanism (DeepSeek Sparse Attention, DSA) to the V3.1 “Terminus” backbone and shows large cost reductions on 128K context without notable quality loss. Model and inference code are released.
DSA design: a tiny FP8 “lightning indexer” scores past tokens per query, then a top-k selector fetches only those KV entries for the main attention. This changes core attention from O(L²) to approximately O(L·k) for the main path while keeping the indexer lightweight.
Training recipe: start from the 128K V3.1 checkpoint. Warm-up with dense attention while training only the indexer via KL to the dense attention distribution (about 2.1B tokens). Switch to sparse training and optimize all weights with k=2048 selected KV tokens per query (≈944B tokens). Post-train with the same pipeline as V3.1 to isolate DSA’s impact.
Post-training stack: specialist distillation for five domains (math, competitive programming, logical reasoning, agentic coding, agentic search) plus writing and QA, then a single mixed RL stage using GRPO to balance reasoning, agent behavior, and alignment. The RL design uses outcome rewards, length penalties, and language-consistency rewards.
Results: quality tracks V3.1 across general, code, search-agent, and math suites. Table 1 shows near-parity on most metrics, with small drops on GPQA/HLE/HMMT that vanish when using checkpoints with similar token lengths. RL curves for BrowseComp and SWE Verified remain stable with DSA.
Cost and latency: The work shows clear end-to-end token-position cost reductions for both prefilling and decoding at long contexts. For short prefills, they provide a masked MHA path to simulate DSA efficiently. Overall effect: significantly cheaper long-context service while preserving accuracy.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack