One-sentence Summary

Researchers from Shanghai Jiao Tong University, Xiaohongshu Inc., and collaborators propose xHC, the first HC-family method to scale beyond $N=4$N=4, which combines temporal feature augmentation with a sparse residual-stream architecture updating only $k=4$k=4 of $N=16$N=16 streams to overcome write-back and cubic mixing bottlenecks, achieving a 4.0-point downstream improvement on $18B$18B MoE models and $1.50\times$1.50× compute savings over vanilla baselines, along with a memory-efficient xHC-Flash variant that reduces per-sublayer memory traffic from $73.5C$73.5C to $40C$40C for practical LLM pre-training.

Key Contributions

• Two bottlenecks are identified that limit scaling Hyper-Connections beyond N=4: insufficient write-back information and cubic-cost residual mixing. Expanded Hyper-Connections (xHC) is introduced, combining temporal feature augmentation with a sparse residual-stream architecture that updates only k=4 of N=16 streams while retaining dense read access.
• Large-scale experiments on 18B and 28B MoE models show xHC improves average downstream scores by 4.0 points over mHC. Scaling-law analyses demonstrate that vanilla and mHC baselines require 1.50× and 1.19× the compute of xHC to reach the same loss.
• xHC-Flash and fused kernels reduce per-sublayer memory traffic from 73.5C to 40C, comparable to mHC at N=4 (34C), while preserving most of xHC's gains. A fused implementation further cuts kernel launch overhead.

Introduction

Large language model architectures have improved through attention mechanisms, Mixture-of-Experts, and scaling, yet the residual stream that carries token representations across layers remains a single identity pathway. Hyper-Connections (HC) and its stabilized variant Manifold-Constrained HC (mHC) introduced multiple parallel residual streams with learnable mixing, showing that expanding from one to four streams yields strong gains and hints at a new scaling axis. However, prior HC-family methods stall at four streams because naively adding more streams brings rapidly diminishing returns: an information bottleneck where a single write-back signal cannot supply enough diversity to populate many streams, and a computational bottleneck where the residual mixing cost grows cubically with the number of streams. The authors propose Expanded Hyper-Connections (xHC), the first method to make residual-stream expansion beyond four streams both effective and affordable. xHC addresses the bottlenecks through temporal feature augmentation, which enriches the write-back signal with multi-scale causal features from neighboring tokens, and a sparse residual-stream architecture that activates only a subset of streams for mixing and write-back while keeping the read path dense. This combination turns expansion rate into a practical scaling dimension, delivering better loss and downstream performance with minimal extra FLOPs.

Method

The authors begin by reviewing Hyper-Connections (HC), which generalize standard residual connections by maintaining $N$N parallel residual streams to increase the capacity of cross-layer information flow. Let $X_l \in \mathbb{R}^{N \times C}$Xl​∈RN×C denote the multi-stream state at layer $l$l, where $C$C is the hidden dimension. HC introduces three learnable mappings per layer: a premapping $\mathcal{H}_l^{\text{pre}}$Hlpre​, a post-mapping $\mathcal{H}_l^{\text{post}}$Hlpost​, and a residual mapping $\mathcal{H}_l^{\text{res}}$Hlres​. The single-layer update is defined as:

$X_{l+1} = \mathcal{H}_l^{\text{res}} X_l + \mathcal{H}_l^{\text{post}} \mathcal{F}(\mathcal{H}_l^{\text{pre}} X_l, \mathcal{W}_l)$Xl+1​=Hlres​Xl​+Hlpost​F(Hlpre​Xl​,Wl​)

To ensure stable training at scale, Manifold-Constrained HC (mHC) projects $\mathcal{H}_l^{\text{res}}$Hlres​ onto the Birkhoff polytope of doubly stochastic matrices via the Sinkhorn-Knopp algorithm. This constraint preserves the identity mapping property across depth.

However, the authors observe that scaling the expansion rate $N$N beyond 4 in mHC quickly saturates. As shown in the figure below, increasing $N$N from 4 to 16 reduces loss by only 0.006 while training FLOPs increase by 32 percent.

The authors identify two bottlenecks behind this saturation. The first is the Information Supply Bottleneck. In mHC, each layer injects only a single write-back component into all streams. As $N$N grows, additional streams become redundant because they cannot draw from diverse components to form non-redundant histories. The second is the Cost Bottleneck. Generating the $N \times N$N×N residual mapping requires predicting $N^2$N2 coefficients from an $NC$NC-dimensional state, leading to an $O(N^3 C)$O(N3C) input-dependent projection cost.

To make large-$N$N expansion both effective and affordable, the authors propose xHC (Expanded Hyper-Connections). Refer to the framework diagram for an overview of the architecture.

xHC employs two coordinated designs: temporal feature augmentation to enrich the write-back signal, and a sparse residual-stream architecture to reduce mixing costs.

Temporal Feature Augmentation

To address the information bottleneck, xHC enriches the write-back basis by borrowing low-cost local contextual information from neighboring tokens. The authors apply $r$r causal depthwise 1D convolutions with kernel sizes $\{\kappa_1, \dots, \kappa_r\}${κ1​,…,κr​} to the layer output. These components capture neighboring-token information at different contextual ranges and are concatenated with the original output:

$\text{out}_{\text{aug}} = [\text{out}; \text{DWConv}_{\kappa_1}(\text{out}); \dots; \text{DWConv}_{\kappa_r}(\text{out})] \in \mathbb{R}^{S \times K_r \times C}$outaug​=[out;DWConvκ1​​(out);…;DWConvκr​​(out)]∈RS×Kr​×C

where $K_r = r + 1$Kr​=r+1. Because depthwise convolutions operate channel-wise, their outputs can retain strong alignment with the original layer output. To prevent redundancy, the authors apply modified Gram-Schmidt orthogonalization over the $K_r$Kr​ components. For $j = 1, \dots, r$j=1,…,r, the orthogonalized components are computed as:

$v_{j+1} = g_j - \sum_{i=1}^j \frac{\langle g_j, v_i \rangle}{\langle v_i, v_i \rangle} v_i$vj+1​=gj​−∑i=1j​⟨vi​,vi​⟩⟨gj​,vi​⟩​vi​

where $g_j = \text{DWConv}_{\kappa_j}(\text{out})$gj​=DWConvκj​​(out) and $v_1 = \text{out}$v1​=out. These orthogonalized components are used for all subsequent write-back operations, ensuring diverse information supply for the expanded streams.

Sparse Residual-Stream Architecture

To address the cost bottleneck, xHC adopts an asymmetric design where only $k$k active streams (e.g., $k=4$k=4) undergo residual mixing and write-back, while dense read access is maintained for all $N$N streams (e.g., $N=16$N=16). The forward pass involves three steps:

1. Stream Routing: A router selects $k$k streams to update at each sublayer. It uses a fixed-plus-routed scheme where $m$m streams are always active with routing weight 1, and the remaining $k-m$k−m streams are selected via TopK routing over sigmoid scores derived from the full $N$N-stream state.
2. Dense Read: Every layer accesses the full $N$N-stream state to form the input: $\text{input}_l = \sum_{i=1}^N h_{l,i}^{\text{pre}} \cdot x_{l,i}$inputl​=∑i=1N​hl,ipre​⋅xl,i​ This preserves cross-layer information flow even though only $k$k streams are updated.
3. Sparse Residual Update: The residual mapping $\mathcal{H}_l^{\text{res}}$Hlres​ and post-mapping $\mathcal{H}_l^{\text{post}}$Hlpost​ are generated only from the active state $X_{\text{active}}$Xactive​. This reduces the dominant generation cost from $O(N^3 C)$O(N3C) to $O(k^3 C)$O(k3C). The active streams are updated as: $X_{\text{active}}^{\text{new}} = \mathcal{H}_l^{\text{res}} X_{\text{active}} + \Delta X_{\text{active}}$Xactivenew​=Hlres​Xactive​+ΔXactive​ where $\Delta X_{\text{active}}$ΔXactive​ combines the augmented write-back components using the post-mapping. Updated streams are scattered back to the full state, while non-active streams carry forward unchanged.

Parameterization and Practical Deployment

The mappings follow a shared gated-projection pattern. A learnable gating scalar $\alpha$α (initialized to 0.01) scales the input-dependent term relative to a learnable bias. The residual mapping uses Sinkhorn normalization on the active streams. The post-mapping uses a $2\sigma(\cdot)$2σ(⋅) scaling to allow both attenuation and mild amplification of individual write-back components.

To further reduce memory traffic during training, the authors introduce xHC-Flash. This lightweight variant amortizes full-state operations across consecutive sublayers. It shares routing decisions and jointly forms sublayer-specific pre-mappings from the block-entry state. By removing residual mixing from the Attention sublayer and deferring it to the MLP side, xHC-Flash enables exact dense-read reuse. This significantly lowers I/O costs to levels comparable to mHC at $N=4$N=4 while retaining the performance advantages of large-$N$N expansion.

Experiment

the paper evaluate xHC in Mixture-of-Experts language model pre-training across scales from 2.5B to 28B parameters, comparing against multi-stream HC (mHC) and a vanilla residual baseline. xHC consistently outperforms both on downstream benchmarks and traces a lower loss curve in scaling-law experiments, yielding a 1.19× compute advantage over mHC at matched loss. Increasing the expansion rate N from 2 to 16 improves loss under xHC with minimal FLOPs overhead, while mHC saturates, demonstrating that sparse updates and temporal feature augmentation effectively address the information and cost bottlenecks of large-N expansion. Ablations confirm that both components are necessary, and xHC remains effective with the Muon optimizer, though its higher memory traffic motivates further efficiency optimizations.

xHC consistently outperforms the vanilla residual baseline and mHC across language understanding and reasoning benchmarks at both 18B and 28B scales. The average score improves from 44.8 with mHC to 48.8 with xHC at 18B, and from 50.5 to 53.6 at 28B, while adding only 3.0% training FLOPs overhead. On MMLU, xHC reaches 57.2 at 18B and 60.5 at 28B, surpassing mHC by 2.5 and 3.7 points respectively. At 18B, xHC boosts BBH score to 39.5, a 5.8-point gain over mHC and a 7.1-point gain over the vanilla baseline.

Adding temporal feature augmentation to multi-head cycling improves validation loss at large expansion rates, confirming the existence of an information bottleneck. The sparse residual-stream architecture then preserves this gain while drastically reducing extra training FLOPs, making large-N states affordable. Dense Read and fixed streams are essential for stable sparse updates, and an active-stream budget of k=4 provides the best performance–cost trade-off. Temporal feature augmentation reduces validation loss from 1.998 to 1.984 for mHC at N=16, showing enriched write-back signals help overcome the information bottleneck. The sparse architecture cuts extra FLOPs from 20.1% to 3.3% while keeping loss nearly unchanged (1.983), making large-N expansion practical. Removing both Dense Read and fixed streams collapses loss to 1.997, exposing the risk of information disconnection when streams are not consistently accessible. Removing fixed streams alone (while keeping Dense Read) increases loss from 1.983 to 1.986, indicating that guaranteed write targets stabilize sparse updates. An active-stream budget of k=4 strikes the best balance: k=2 yields worse loss (1.991), while k=8 gives only marginal improvement (1.982) at higher cost.

On an 18B MoE model, xHC combined with the Muon optimizer (without Gram–Schmidt orthogonalization) consistently outperforms both the AdamW and Muon baselines across language understanding and reasoning tasks. The Muon baseline itself improves over AdamW, and adding xHC yields further substantial gains, particularly on CommonsenseQA and BBH, demonstrating xHC's compatibility with Muon and its effectiveness beyond the AdamW optimizer. Muon + xHC lifts CommonsenseQA from 49.1 to 61.3, the largest relative improvement among the evaluated benchmarks. Across all five benchmarks, xHC adds consistent gains over the Muon baseline, with MMLU rising from 51.3 to 56.6 and BBH from 36.1 to 42.2.

xHC increases per-sublayer memory traffic relative to mHC, mainly due to two full-state reads over its larger residual representation at every sublayer. The xHC-Flash variant amortizes these operations across consecutive sublayers, bringing traffic down to levels comparable to mHC while retaining most of xHC's performance gains. With N=16 and k=4, xHC requires roughly 2.2× the per-sublayer memory traffic of mHC at its standard N=4 setting. xHC-Flash reduces per-sublayer traffic from 73.5C to 51C by sharing routing and pre-mappings within a block; its four-sublayer extension further drops traffic to 40C, approaching mHC's 34C.

xHC-Flash reduces the memory traffic of the full xHC method by roughly 30% while preserving its validation loss. Its four-sublayer extension further cuts traffic to a level close to that of mHC, yet retains a clear loss advantage over mHC. xHC-Flash matches the full xHC validation loss of 1.983 while lowering amortized per-sublayer I/O from 73.5C to 51C. xHC-Flash-4sub achieves 40C I/O, approaching mHC's 34C, and its loss of 1.984 remains substantially better than mHC's 2.004.

The experiments evaluate xHC against vanilla residual and mHC baselines on language understanding and reasoning benchmarks, demonstrating consistent performance gains at 18B and 28B scales with only a 3% training FLOPs overhead. Ablations confirm that temporal feature augmentation mitigates an information bottleneck in multi-head cycling, and a sparse residual-stream design with Dense Read and fixed streams drastically reduces extra FLOPs while preserving loss, with an active-stream budget of k=4 offering the best trade-off. xHC also proves compatible with the Muon optimizer, yielding further improvements on an 18B MoE model. Finally, the xHC-Flash variant amortizes memory traffic to levels near mHC while retaining the performance benefits, making the method practical for large-scale deployment.