HyperAIHyperAI

Command Palette

Search for a command to run...

Multi-Branch Transformer

Gaotang Li Lingjie Chen Chunyuan Deng Zhiwei Xu Hongzheng Chen Christopher De Sa Jingrui He Yingheng Wang Hanghang Tong

Zusammenfassung

Transformer-Sprachmodelle werden üblicherweise skaliert, indem ihre Blöcke verbreitert oder tiefer gestapelt werden. In dieser Arbeit untersuchen wir einen komplementären Weg: das Branching, die parallele Zusammensetzung vollständiger Transformer-Blöcke. Wir stellen den Multi-Branch Transformer (MBT) vor, der die Eingabe durch gemeinsame initiale Schichten verarbeitet, unabhängig parametrisierte Zweige sequenzieller Blöcke parallel ausführt und deren Ausgaben fusioniert, bevor das Ergebnis durch gemeinsame finale Schichten geleitet wird. Bei vergleichbarem Trainingsaufwand im Verhältnis zu einem Standard-Transformer untersuchen wir drei MBT-Varianten: Wide-Shallow erhält die Blockbreite, Narrow-Deep erhält die sequenzielle Tiefe, und Balanced interpoliert zwischen diesen Designs. In kontrollierten Pretraining-Experimenten auf vier Mixture-of-Experts-Modellskalen mit insgesamt 0,8 bis 2,1 Milliarden Parametern zeigen wir, dass 2-zweigige Wide-Shallow-MBTs sequenzielle Transformer bei gleichen Parameterund Trainingsbudgets durchgängig übertreffen. Balanced-MBTs verbessern die Modellierungsleistung gegenüber ihren Wide-Shallow-Pendants weiter. Diese Modelle behalten eine vergleichbare Qualität, wenn die Anzahl der Zweige von 2 auf 3 oder 4 steigt. Um die Inferenzeffizienz bei langen Kontexten weiter zu verbessern, führen wir Joint-KV ein, das einen gemeinsamen KV-Cache aus den Zweigrepräsentationen auf jeder Tiefe konstruiert. Unser finales Modellrezept kombiniert Balanced-MBTs mit Joint-KV, unterstützt durch eine Inferenz-Engine, die Zweiggruppierung und Wiederverwendung gemeinsamer Caches nutzt. Angepasste Skalierungskurven zeigen, dass Standard-Transformer bis zu 1,26-mal so viel Trainingsrechenaufwand benötigen, um den Validierungsverlust unserer finalen Modelle zu erreichen. Über die evaluierten Konfigurationen hinweg reduzieren diese Modelle den KV-Cache-Speicherbedarf des Gesamtmodells um bis zu 34 % und liefern bis zu 51,4 % höheren Dekodierungsdurchsatz auf einer einzelnen H200-GPU. Umfangreiche Ablationsstudien zeigen, dass Branching Gewinne über Änderungen von Breite und Tiefe hinaus erzielt, eine effektive KV-Sharing-Strategie (Joint-KV) ermöglicht, robust gegenüber Fusionsfunktionen bleibt und in den mittleren Schichten am besten funktioniert. Zusammen belegen diese Ergebnisse, dass Branching einen praktischen Weg zur Skalierung von Sprachmodellen bietet – hin zu besserer Modellierungsqualität und schnellerer Inferenz.

One-sentence Summary

Researchers at the University of Illinois Urbana-Champaign, Rice University, and colleagues propose the Multi-Branch Transformer (MBT), which processes inputs through shared initial layers, runs independently parameterized parallel Transformer blocks, and fuses outputs before shared final layers; its Balanced variant with Joint-KV shared caching outperforms sequential Transformers under matched parameter and training-compute budgets, reducing KV-cache storage by up to 34%34\%34% and increasing decoding throughput by up to 51.4%51.4\%51.4%.

Key Contributions

  • This work introduces the Multi-Branch Transformer (MBT), which processes input through shared initial layers, executes independently parameterized sequential Transformer block branches in parallel, and fuses their outputs before shared final layers; it also defines Wide-Shallow, Narrow-Deep, and Balanced branch allocation variants.
  • The paper proposes Joint-KV, which constructs a shared KV cache from parallel branch representations at each depth while retaining branch-specific queries, and combines it with an inference engine using branch grouping and shared-cache reuse to reduce full-model KV-cache storage by up to 34% and raise single-GPU H200 decoding throughput by up to 51.4%.
  • Controlled pretraining experiments on mixture-of-experts models from 0.8B to 2.1B parameters show that 2-branch Wide-Shallow MBTs outperform matched sequential Transformers under equal parameter and training-compute budgets, Balanced MBTs improve further, and fitted scaling curves indicate standard Transformers require up to 1.26 times as much training body compute to match the final models' validation loss.

Introduction

Scaling Transformer language models usually relies on depth and width choices, while recent architectures expand attention, sparse expert computation, and residual design but still depend on sequential stacks of full blocks. Prior work explores parallelism inside blocks, width-depth tradeoffs, or systems-level execution, leaving the parallel composition of full Transformer blocks underexplored. The authors introduce the Multi-Branch Transformer, which organizes full blocks into parallel branches between shared initial and final layers, and Joint-KV, which shares a key-value cache across branches at the same depth to improve long-context inference efficiency while preserving model quality.

Method

The authors introduce the Multi-Branch Transformer (MBT), which enables parallel computation at the granularity of full Transformer blocks.

An MBT processes the input through shared initial layers (the prelude), splits computation across KKK independently parameterized branches that execute concurrently, fuses their outputs, and passes the result through shared final layers (the coda). The key idea is to preserve sequential composition within each branch while allowing subsequent shared layers to build on the results of multiple transformation paths.

In comparison to related work that explores concurrency within individual blocks, allocation of width and depth, or alternative parallel computation paths, MBTs explore an orthogonal axis of parallelism at the granularity of full Transformer blocks.

Let x=(x1,…,xT)\boldsymbol{x} = (x_1, \ldots, x_T)x=(x1​,…,xT​) be an input token sequence and H∈RT×dH \in \mathbb{R}^{T \times d}H∈RT×d its hidden states at a block's input, where ddd is the residual width. Each block Tℓ\mathcal{T}_\ellTℓ​ applies causal attention followed by a mixture-of-experts (MoE) feed-forward module:

H′=H+Attnℓ(H),Tℓ(H)=H′+MoEℓ(H′).H' = H + \mathrm{Attn}_\ell(H), \qquad \mathcal{T}_\ell(H) = H' + \mathrm{MoE}_\ell(H').H′=H+Attnℓ​(H),Tℓ​(H)=H′+MoEℓ​(H′).

An MBT replaces the serial middle stack with K≥2K \geq 2K≥2 independently parameterized branch stacks Gi\mathcal{G}_iGi​. The prelude P\mathcal{P}P, fusion F\mathcal{F}F, and coda connect them:

H=P(X),Zi=Gi(H),Y^=Dec(C(F(Z1,…,ZK))).H = \mathcal{P}(X), \quad Z_i = \mathcal{G}_i(H), \quad \widehat{Y} = \mathrm{Dec}\big(\mathcal{C}(\mathcal{F}(Z_1, \dots, Z_K))\big).H=P(X),Zi​=Gi​(H),Y=Dec(C(F(Z1​,…,ZK​))).

For fusion, the authors normalize the branch outputs, concatenate them along the hidden dimension, and project the result back to the shared width ddd:

F(Z1,…,ZK)=Concat⁡(N1(Z1),…,NK(ZK))WF.\mathcal{F}(Z_1, \dots, Z_K) = \operatorname{Concat}\bigl(\mathcal{N}_1(Z_1), \dots, \mathcal{N}_K(Z_K)\bigr) W_F.F(Z1​,…,ZK​)=Concat(N1​(Z1​),…,NK​(ZK​))WF​.

To define MBTs at comparable compute, the authors propose three branch width and depth allocations.

Wide-Shallow MBTs preserve the original block width and redistribute the middle blocks across parallel branches. Narrow-Deep MBTs preserve the original sequential depth along each branch while reducing block width. Balanced MBTs interpolate between these choices by jointly adjusting branch width and depth.

To further exploit the parallel branch structure, the authors propose Joint-KV, which constructs a shared key-value (KV) cache from all branch representations at each depth while retaining separate queries for each branch, alongside an inference engine based on nano-vLLM.

In the default Independent-KV configuration, each block in each branch maintains a separate KV cache. In Joint-KV, all branches share one KV cache at each depth, constructed jointly from their pre-attention inputs. Formally, let Sˉi,ℓ∈RT×db\bar{S}_{i, \ell} \in \mathbb{R}^{T \times d_b}Sˉi,ℓ​∈RT×db​ denote the RMS-normalized input to branch iii at depth ℓ\ellℓ. The KV-cache constructions are:

KVi,ℓind=Sˉi,ℓWi,ℓind,KVℓshared=1K[Sˉ1,ℓ;… ;SˉK,ℓ]Wℓshared.\mathrm{KV}_{i, \ell}^{\mathrm{ind}} = \bar{S}_{i, \ell} W_{i, \ell}^{\mathrm{ind}}, \qquad \mathrm{KV}_{\ell}^{\mathrm{shared}} = \frac{1}{\sqrt{K}} [\bar{S}_{1, \ell}; \dots; \bar{S}_{K, \ell}] W_{\ell}^{\mathrm{shared}}.KVi,ℓind​=Sˉi,ℓ​Wi,ℓind​,KVℓshared​=K​1​[Sˉ1,ℓ​;…;SˉK,ℓ​]Wℓshared​.

At fixed branch depth and KV dimensions, the Joint-KV cache in the branch region is 1/K1/K1/K that of Independent-KV.

The inference engine employs two additional execution mechanisms. Branch grouping groups blocks at the same depth across branches into a batched execution stage. With Joint-KV, the engine first constructs the shared KV cache from the branch input representations, then computes the branch-specific attention and MoE outputs together. This reduces the number of sequential block stages. Head folding folds the branch dimension into the query-head dimension, grouping queries by their shared KV head so that all branches use a single grouped-query attention (GQA) call. This allows the attention kernel to reuse shared KV data across branches, reducing redundant memory traffic while preserving the attention function.

Experiment

The experiments pretrain standard Transformers and multi-branch Transformers across a four-scale ladder on FineWeb-Edu, systematically varying branch count, width-depth allocation, KV sharing, fusion, and branch placement. Allocating roughly one third of blocks to two parallel branches consistently improves validation loss and zero-shot downstream performance, and the benefit persists with three or four branches, especially with balanced width-depth tuning. Joint-KV preserves these quality gains while substantially reducing KV cache size and improving decoding throughput over standard Transformers. Ablations show that branching helps beyond simple widening or depth matching, and that shared KV, concat-projection fusion, and middle branch placement are the most effective choices.

Across four model scales, the selected Wide-Shallow 2-branch MBTs keep total and active-body parameters nearly identical to standard Transformers while achieving lower validation loss at every scale. The best models place a minority of physical blocks in two equal parallel branches, with equal-depth prelude and coda, and parallel block ratios cluster around one third. Loss reductions are modest, ranging from about 0.025 to 0.031. Selected MBTs reduce validation loss at every scale while keeping total and active-body parameters almost unchanged relative to standard Transformers. The best allocations consistently place roughly one third of physical blocks in two equal parallel branches, with prelude and coda depths matched in each selected configuration. Validation loss improvements are slightly larger at smaller scales, decreasing from about 0.031 to 0.025 as model scale increases.

At this matched-compute setting, multi-branch transformers with two, three, or four parallel branches consistently improve validation loss over the serial baseline. Balanced width-depth allocations, which tune block width and total depth to preserve body compute, generally reach lower loss than Wide-Shallow allocations that keep the base width and spread a fixed block budget across branches. Increasing branch count from two to four produces only modest loss changes while preserving the multi-branch advantage. Every multi-branch configuration shown improves over the serial baseline at comparable body compute. Balanced allocations achieve lower validation loss than Wide-Shallow allocations at each branch count. Balanced models use narrower blocks and more total physical blocks, while Wide-Shallow models keep the base width and reduce depth as branches increase.

At the W4 scale, Balanced MBTs with 2, 3, and 4 branches improve validation loss over the Standard model while using less KV cache per token. Joint-KV variants match their Independent-KV counterparts in loss and provide growing cache savings relative to Standard as branch count increases. The results show that shared KV across branches can retain language modeling quality while improving cache efficiency. All W4 MBT variants achieve lower validation loss than Standard while also reducing KV cache size. Joint-KV cache savings relative to Standard increase with branch count, reaching about 32% at four branches, while loss remains nearly flat across configurations.

The ablation compares branching models with sequential widening and depth-matched controls at roughly equal body compute. Wide-Shallow MBT outperforms both global and local widening baselines, while Narrow-Deep MBT outperforms Standard and shows the steepest scaling exponent. These patterns indicate that branching improves modeling quality beyond shallower or wider sequential alternatives, and that width and depth can be jointly tuned. Neither global widening nor core-only widening reproduces the gains of the branching model at matched compute. A depth-matched narrow branching model outperforms the standard sequential model and achieves the steepest scaling trend.

Across matched KV-cache comparisons at all four scales, Joint-KV consistently achieves lower validation loss than Independent-KV. The improvement is similar across scales, while Joint-KV's larger KV projection adds only a small amount of body compute. This indicates that sharing keys and values across branches is more effective than maintaining smaller private KV caches of equal total size. Joint-KV lowers validation loss at every tested scale, by 0.011 to 0.012. Joint-KV's larger KV projection adds less than 0.6 percent body compute, far too little to explain the loss improvement.

Across four model scales and matched-compute settings, multi-branch transformers with two to four parallel branches consistently improve validation loss over serial baselines while keeping total and active-body parameters nearly unchanged, and balanced width-depth allocations generally outperform wide-shallow allocations. Ablations show that these gains come from branching itself rather than from simple widening or depth scaling. Shared Joint-KV cache variants retain or slightly improve language modeling quality while reducing cache use, with savings relative to standard models growing to about 32 percent at four branches, and they consistently outperform equal-size independent KV caches at every tested scale.


KI mit KI entwickeln

Von der Idee bis zum Launch – beschleunigen Sie Ihre KI-Entwicklung mit kostenlosem KI-Co-Coding, sofort einsatzbereiter Umgebung und bestem GPU-Preis.

KI-gestütztes kollaboratives Programmieren
Sofort einsatzbereite GPUs
Die besten Preise

HyperAI Newsletters

Abonnieren Sie unsere neuesten Updates
Wir werden die neuesten Updates der Woche in Ihren Posteingang liefern um neun Uhr jeden Montagmorgen
Unterstützt von MailChimp