How Does mHC Use Its Residual Streams? Selective Routing and Near-Identity Mixing
Pengxiang Zhao and colleagues at Huawei measure how DeepSeek-V4-Flash actually uses the four-stream residual pathway that manifold-constrained Hyper-Connections provide, and find most of the added capacity unused.
Ask this paper
Read/write routing is narrow: A typical attention or FFN site effectively uses about two of the four streams. The dominant stream changes across depth and stream representations stay directionally distinct.
Mixing is front-loaded: Cross-stream residual mixing is modest and concentrated in early layers. In layers 22 to 42 the pathway mostly carries each stream forward separately.
Interventions confirm it: Replacing late mixers with identity raises C4 perplexity by only 1.9% and preserves the six-task average, while replacing early mixers raises perplexity by 41%.
Early mixers matter structurally, not token-wise: Fixing each early mixer to its C4 diagnostic mean costs 0.2% perplexity and 0.25 points of average score, so the site-specific structure carries the value rather than the per-token variation. Keeping only the three largest routing weights per token costs at most 2.7% perplexity and 0.4 points.
Abstract
Hyper-Connections and their manifold-constrained variant mHC widen a residual pathway from one stream to n, yet how trained models use this capacity remains unclear: how broadly blocks read and write, how strongly the residual pathway mixes streams, and whether the streams carry distinct representations. We examine these properties in the four-stream residual pathway of DeepSeek-V4-Flash using effective stream counts, cross-stream residual weights, and inter-stream cosine similarity. Read/write routing is concentrated but varies across depth: a typical attention or FFN site effectively uses about two streams, while the dominant stream changes across layers and the representations remain directionally distinct. Residual mixing is modest and occurs primarily in early layers; in layers 22-42, the pathway mostly carries each stream forward separately. Targeted interventions establish the functional significance of these patterns. Replacing the late mixers by identity increases C4 perplexity by only 1.9% and preserves the six-task average score, whereas replacing the early mixers increases perplexity by 41%. Fixing each early mixer to its C4 diagnostic mean increases perplexity by only 0.2% and reduces the average score by 0.25 percentage points, showing that its site-specific structure matters more than its token-wise variation on the evaluated metrics. Likewise, retaining the three largest routing weights per token at every site increases perplexity by at most 2.7% and changes the average score by at most 0.4 points. Thus, the studied model realizes only part of the flexibility afforded by four-stream mHC: individual blocks rarely require all four streams, and late residual mixing provides little measured benefit.