Manifold-Constrained Hyper-Connections
Free while signed in. Answers cite the passages they came from.
This DeepSeek paper proposes Manifold-Constrained Hyper-Connections (mHC), a framework that extends residual connections by expanding residual stream width while restoring training stability. The key insight: unconstrained Hyper-Connections compromise identity mapping, causing training instability at scale. - **Identity mapping restoration:** mHC projects residual connection matrices onto the Birkhoff polytope using the Sinkhorn-Knopp algorithm, constraining them to doubly stochastic matrices. This preserves feature mean during propagation and prevents vanishing or exploding signals. - **Stability at scale:** Standard Hyper-Connections showed loss surges around 12k steps with gradient norm instability and Amax Gain Magnitude peaks of 3000. mHC maintains stable training by ensuring the composite mapping across layers preserves conservation properties. - **Efficient infrastructure:** The approach uses kernel fusion with TileLang, selective recomputing to reduce memory footprint, and overlapped communication within DualPipe schedule. This introduces only 6.7% additional time overhead at expansion rate n=4. - **Scalable performance:** Experiments demonstrate mHC maintains the performance advantages of Hyper-Connections while enabling training at scale, offering a practical path for scaling via residual stream width rather than just model FLOPs or data size.
Identity mapping restoration: mHC projects residual connection matrices onto the Birkhoff polytope using the Sinkhorn-Knopp algorithm, constraining them to doubly stochastic matrices. This preserves the feature mean during propagation and prevents vanishing or exploding signals.
Stability at scale: Standard Hyper-Connections showed loss surges around 12k steps with gradient norm instability and Amax Gain Magnitude peaks of 3000. mHC maintains stable training by ensuring the composite mapping across layers preserves conservation properties.
Efficient infrastructure: The approach uses kernel fusion with TileLang, selective recomputing to reduce memory footprint, and overlapped communication within the DualPipe schedule. This introduces only 6.7% additional time overhead at expansion rate n=4.
Scalable performance: Experiments demonstrate mHC maintains the performance advantages of Hyper-Connections while enabling training at scale, offering a practical path for scaling via residual stream width rather than just model FLOPs or data size.
Abstract
Recently, studies exemplified by Hyper-Connections (HC) have extended the ubiquitous residual connection paradigm established over the past decade by expanding the residual stream width and diversifying connectivity patterns. While yielding substantial performance gains, this diversification fundamentally compromises the identity mapping property intrinsic to the residual connection, which causes severe training instability and restricted scalability, and additionally incurs notable memory access overhead. To address these challenges, we propose Manifold-Constrained Hyper-Connections (mHC), a general framework that projects the residual connection space of HC onto a specific manifold to restore the identity mapping property, while incorporating rigorous infrastructure optimization to ensure efficiency. Empirical experiments demonstrate that mHC is effective for training at scale, offering tangible performance improvements and superior scalability. We anticipate that mHC, as a flexible and practical extension of HC, will contribute to a deeper understanding of topological architecture design and suggest promising directions for the evolution of foundational models.
Get next week’s papers.
The same picks and the same summaries, in your inbox. Free, and 176 issues deep.
Subscribe on Substack