HyperAIHyperAI

Command Palette

Search for a command to run...

マルチブランチTransformer

Gaotang Li Lingjie Chen Chunyuan Deng Zhiwei Xu Hongzheng Chen Christopher De Sa Jingrui He Yingheng Wang Hanghang Tong

概要

Transformer言語モデルは、通常、ブロックを広くするか、より深く積み重ねることでスケールされる。本研究では、補完的な経路として、完全なTransformerブロックの並列合成である「分岐」を探求する。我々は、共有初期層を通じて入力を処理し、独立にパラメータ化された逐次ブロックの分岐を並列に実行し、その出力を融合してから共有最終層に渡す、マルチブランチTransformer(MBT)を導入する。標準的なTransformerと同等の訓練計算量において、3つのMBT変種を検討する。Wide-Shallowはブロック幅を維持し、Narrow-Deepは逐次深さを維持し、Balancedはこれらの設計を補間する。総パラメータ数0.8Bから2.1Bにわたる4つのMixture-of-Expertsモデル規模での制御された事前学習実験を通じて、2分岐のWide-Shallow MBTが、一致したパラメータ数と訓練計算量の予算の下で、逐次Transformerを一貫して上回ることを示す。Balanced MBTは、Wide-Shallowの対応モデルよりもモデリング性能をさらに向上させる。これらのモデルは、分岐数が2から3または4に増加しても同等の品質を維持する。長い文脈での推論効率をさらに向上させるため、各深さで分岐表現から共有KVキャッシュを構築するJoint-KVを導入する。最終的なモデルレシピは、Balanced MBTとJoint-KVを組み合わせ、分岐グループ化と共有キャッシュ再利用を活用する推論エンジンによって支えられる。適合させたスケーリング曲線は、標準的なTransformerが最終モデルの検証損失に匹敵するために最大1.26倍の訓練本体計算量を必要とすることを示す。評価した構成全体で、これらのモデルはフルモデルのKVキャッシュ保存量を最大34%削減し、単一のH200 GPU上で最大51.4%高いデコードスループットを実現する。広範なアブレーション研究により、分岐は幅と深さの変更を超えた利得をもたらし、効果的なKV共有戦略(Joint-KV)を可能にし、融合関数に対して頑健であり、中間層で最も優れた性能を発揮することが示される。これらの知見は、分岐が言語モデルをより優れたモデリング品質とより高速な推論の両方へスケールするための実用的な道を提供することを示している。

One-sentence Summary

Researchers at the University of Illinois Urbana-Champaign, Rice University, and colleagues propose the Multi-Branch Transformer (MBT), which processes inputs through shared initial layers, runs independently parameterized parallel Transformer blocks, and fuses outputs before shared final layers; its Balanced variant with Joint-KV shared caching outperforms sequential Transformers under matched parameter and training-compute budgets, reducing KV-cache storage by up to 34%34\%34% and increasing decoding throughput by up to 51.4%51.4\%51.4%.

Key Contributions

  • This work introduces the Multi-Branch Transformer (MBT), which processes input through shared initial layers, executes independently parameterized sequential Transformer block branches in parallel, and fuses their outputs before shared final layers; it also defines Wide-Shallow, Narrow-Deep, and Balanced branch allocation variants.
  • The paper proposes Joint-KV, which constructs a shared KV cache from parallel branch representations at each depth while retaining branch-specific queries, and combines it with an inference engine using branch grouping and shared-cache reuse to reduce full-model KV-cache storage by up to 34% and raise single-GPU H200 decoding throughput by up to 51.4%.
  • Controlled pretraining experiments on mixture-of-experts models from 0.8B to 2.1B parameters show that 2-branch Wide-Shallow MBTs outperform matched sequential Transformers under equal parameter and training-compute budgets, Balanced MBTs improve further, and fitted scaling curves indicate standard Transformers require up to 1.26 times as much training body compute to match the final models' validation loss.

Introduction

Scaling Transformer language models usually relies on depth and width choices, while recent architectures expand attention, sparse expert computation, and residual design but still depend on sequential stacks of full blocks. Prior work explores parallelism inside blocks, width-depth tradeoffs, or systems-level execution, leaving the parallel composition of full Transformer blocks underexplored. The authors introduce the Multi-Branch Transformer, which organizes full blocks into parallel branches between shared initial and final layers, and Joint-KV, which shares a key-value cache across branches at the same depth to improve long-context inference efficiency while preserving model quality.

Method

The authors introduce the Multi-Branch Transformer (MBT), which enables parallel computation at the granularity of full Transformer blocks.

An MBT processes the input through shared initial layers (the prelude), splits computation across KKK independently parameterized branches that execute concurrently, fuses their outputs, and passes the result through shared final layers (the coda). The key idea is to preserve sequential composition within each branch while allowing subsequent shared layers to build on the results of multiple transformation paths.

In comparison to related work that explores concurrency within individual blocks, allocation of width and depth, or alternative parallel computation paths, MBTs explore an orthogonal axis of parallelism at the granularity of full Transformer blocks.

Let x=(x1,…,xT)\boldsymbol{x} = (x_1, \ldots, x_T)x=(x1​,…,xT​) be an input token sequence and H∈RT×dH \in \mathbb{R}^{T \times d}H∈RT×d its hidden states at a block's input, where ddd is the residual width. Each block Tℓ\mathcal{T}_\ellTℓ​ applies causal attention followed by a mixture-of-experts (MoE) feed-forward module:

H′=H+Attnℓ(H),Tℓ(H)=H′+MoEℓ(H′).H' = H + \mathrm{Attn}_\ell(H), \qquad \mathcal{T}_\ell(H) = H' + \mathrm{MoE}_\ell(H').H′=H+Attnℓ​(H),Tℓ​(H)=H′+MoEℓ​(H′).

An MBT replaces the serial middle stack with K≥2K \geq 2K≥2 independently parameterized branch stacks Gi\mathcal{G}_iGi​. The prelude P\mathcal{P}P, fusion F\mathcal{F}F, and coda connect them:

H=P(X),Zi=Gi(H),Y^=Dec(C(F(Z1,…,ZK))).H = \mathcal{P}(X), \quad Z_i = \mathcal{G}_i(H), \quad \widehat{Y} = \mathrm{Dec}\big(\mathcal{C}(\mathcal{F}(Z_1, \dots, Z_K))\big).H=P(X),Zi​=Gi​(H),Y=Dec(C(F(Z1​,…,ZK​))).

For fusion, the authors normalize the branch outputs, concatenate them along the hidden dimension, and project the result back to the shared width ddd:

F(Z1,…,ZK)=Concat⁡(N1(Z1),…,NK(ZK))WF.\mathcal{F}(Z_1, \dots, Z_K) = \operatorname{Concat}\bigl(\mathcal{N}_1(Z_1), \dots, \mathcal{N}_K(Z_K)\bigr) W_F.F(Z1​,…,ZK​)=Concat(N1​(Z1​),…,NK​(ZK​))WF​.

To define MBTs at comparable compute, the authors propose three branch width and depth allocations.

Wide-Shallow MBTs preserve the original block width and redistribute the middle blocks across parallel branches. Narrow-Deep MBTs preserve the original sequential depth along each branch while reducing block width. Balanced MBTs interpolate between these choices by jointly adjusting branch width and depth.

To further exploit the parallel branch structure, the authors propose Joint-KV, which constructs a shared key-value (KV) cache from all branch representations at each depth while retaining separate queries for each branch, alongside an inference engine based on nano-vLLM.

In the default Independent-KV configuration, each block in each branch maintains a separate KV cache. In Joint-KV, all branches share one KV cache at each depth, constructed jointly from their pre-attention inputs. Formally, let Sˉi,ℓ∈RT×db\bar{S}_{i, \ell} \in \mathbb{R}^{T \times d_b}Sˉi,ℓ​∈RT×db​ denote the RMS-normalized input to branch iii at depth ℓ\ellℓ. The KV-cache constructions are:

KVi,ℓind=Sˉi,ℓWi,ℓind,KVℓshared=1K[Sˉ1,ℓ;… ;SˉK,ℓ]Wℓshared.\mathrm{KV}_{i, \ell}^{\mathrm{ind}} = \bar{S}_{i, \ell} W_{i, \ell}^{\mathrm{ind}}, \qquad \mathrm{KV}_{\ell}^{\mathrm{shared}} = \frac{1}{\sqrt{K}} [\bar{S}_{1, \ell}; \dots; \bar{S}_{K, \ell}] W_{\ell}^{\mathrm{shared}}.KVi,ℓind​=Sˉi,ℓ​Wi,ℓind​,KVℓshared​=K​1​[Sˉ1,ℓ​;…;SˉK,ℓ​]Wℓshared​.

At fixed branch depth and KV dimensions, the Joint-KV cache in the branch region is 1/K1/K1/K that of Independent-KV.

The inference engine employs two additional execution mechanisms. Branch grouping groups blocks at the same depth across branches into a batched execution stage. With Joint-KV, the engine first constructs the shared KV cache from the branch input representations, then computes the branch-specific attention and MoE outputs together. This reduces the number of sequential block stages. Head folding folds the branch dimension into the query-head dimension, grouping queries by their shared KV head so that all branches use a single grouped-query attention (GQA) call. This allows the attention kernel to reuse shared KV data across branches, reducing redundant memory traffic while preserving the attention function.

Experiment

The experiments pretrain standard Transformers and multi-branch Transformers across a four-scale ladder on FineWeb-Edu, systematically varying branch count, width-depth allocation, KV sharing, fusion, and branch placement. Allocating roughly one third of blocks to two parallel branches consistently improves validation loss and zero-shot downstream performance, and the benefit persists with three or four branches, especially with balanced width-depth tuning. Joint-KV preserves these quality gains while substantially reducing KV cache size and improving decoding throughput over standard Transformers. Ablations show that branching helps beyond simple widening or depth matching, and that shared KV, concat-projection fusion, and middle branch placement are the most effective choices.

Across four model scales, the selected Wide-Shallow 2-branch MBTs keep total and active-body parameters nearly identical to standard Transformers while achieving lower validation loss at every scale. The best models place a minority of physical blocks in two equal parallel branches, with equal-depth prelude and coda, and parallel block ratios cluster around one third. Loss reductions are modest, ranging from about 0.025 to 0.031. Selected MBTs reduce validation loss at every scale while keeping total and active-body parameters almost unchanged relative to standard Transformers. The best allocations consistently place roughly one third of physical blocks in two equal parallel branches, with prelude and coda depths matched in each selected configuration. Validation loss improvements are slightly larger at smaller scales, decreasing from about 0.031 to 0.025 as model scale increases.

At this matched-compute setting, multi-branch transformers with two, three, or four parallel branches consistently improve validation loss over the serial baseline. Balanced width-depth allocations, which tune block width and total depth to preserve body compute, generally reach lower loss than Wide-Shallow allocations that keep the base width and spread a fixed block budget across branches. Increasing branch count from two to four produces only modest loss changes while preserving the multi-branch advantage. Every multi-branch configuration shown improves over the serial baseline at comparable body compute. Balanced allocations achieve lower validation loss than Wide-Shallow allocations at each branch count. Balanced models use narrower blocks and more total physical blocks, while Wide-Shallow models keep the base width and reduce depth as branches increase.

At the W4 scale, Balanced MBTs with 2, 3, and 4 branches improve validation loss over the Standard model while using less KV cache per token. Joint-KV variants match their Independent-KV counterparts in loss and provide growing cache savings relative to Standard as branch count increases. The results show that shared KV across branches can retain language modeling quality while improving cache efficiency. All W4 MBT variants achieve lower validation loss than Standard while also reducing KV cache size. Joint-KV cache savings relative to Standard increase with branch count, reaching about 32% at four branches, while loss remains nearly flat across configurations.

The ablation compares branching models with sequential widening and depth-matched controls at roughly equal body compute. Wide-Shallow MBT outperforms both global and local widening baselines, while Narrow-Deep MBT outperforms Standard and shows the steepest scaling exponent. These patterns indicate that branching improves modeling quality beyond shallower or wider sequential alternatives, and that width and depth can be jointly tuned. Neither global widening nor core-only widening reproduces the gains of the branching model at matched compute. A depth-matched narrow branching model outperforms the standard sequential model and achieves the steepest scaling trend.

Across matched KV-cache comparisons at all four scales, Joint-KV consistently achieves lower validation loss than Independent-KV. The improvement is similar across scales, while Joint-KV's larger KV projection adds only a small amount of body compute. This indicates that sharing keys and values across branches is more effective than maintaining smaller private KV caches of equal total size. Joint-KV lowers validation loss at every tested scale, by 0.011 to 0.012. Joint-KV's larger KV projection adds less than 0.6 percent body compute, far too little to explain the loss improvement.

Across four model scales and matched-compute settings, multi-branch transformers with two to four parallel branches consistently improve validation loss over serial baselines while keeping total and active-body parameters nearly unchanged, and balanced width-depth allocations generally outperform wide-shallow allocations. Ablations show that these gains come from branching itself rather than from simple widening or depth scaling. Shared Joint-KV cache variants retain or slightly improve language modeling quality while reducing cache use, with savings relative to standard models growing to about 32 percent at four branches, and they consistently outperform equal-size independent KV caches at every tested scale.


AIでAIを構築

アイデアからローンチまで — 無料のAIコーディング支援、すぐに使える環境、最高のGPU価格でAI開発を加速。

AI コーディング補助
すぐに使える GPU
最適な料金体系

HyperAI Newsletters

最新情報を購読する
北京時間 毎週月曜日の午前9時 に、その週の最新情報をメールでお届けします
メール配信サービスは MailChimp によって提供されています