HyperAIHyperAI

Command Palette

Search for a command to run...

멀티브랜치 트랜스포머

Gaotang Li Lingjie Chen Chunyuan Deng Zhiwei Xu Hongzheng Chen Christopher De Sa Jingrui He Yingheng Wang Hanghang Tong

초록

트랜스포머 언어 모델은 일반적으로 블록을 더 넓게 만들거나 더 깊게 쌓는 방식으로 확장된다. 본 연구에서는 전체 트랜스포머 블록을 병렬로 구성하는 브랜칭이라는 보완적 경로를 탐구한다. 입력을 공유 초기 계층으로 처리하고, 독립적으로 매개변수화된 순차 블록의 브랜치들을 병렬로 실행한 뒤, 그 출력을 융합하여 공유 최종 계층에 전달하는 멀티브랜치 트랜스포머(MBT)를 소개한다. 표준 트랜스포머와 유사한 학습 연산량에서 세 가지 MBT 변형을 연구한다. Wide-Shallow는 블록 폭을 유지하고, Narrow-Deep은 순차 깊이를 유지하며, Balanced는 이 두 설계 사이를 보간한다. 총 매개변수 0.8B에서 2.1B에 이르는 네 가지 혼합 전문가 모델 규모에서 수행한 통제된 사전학습 실험을 통해, 매개변수와 학습 연산량 예산이 일치하는 조건에서 2-브랜치 Wide-Shallow MBT가 순차 트랜스포머를 일관되게 능가함을 보인다. Balanced MBT는 Wide-Shallow 대비 모델링 성능을 추가로 향상시킨다. 이 모델들은 브랜치 수가 2개에서 3개 또는 4개로 증가해도 비슷한 품질을 유지한다. 긴 문맥에서 추론 효율을 더욱 개선하기 위해, 각 깊이에서 브랜치 표현으로부터 공유 KV 캐시를 구성하는 Joint-KV를 도입한다. 최종 모델 레시피는 Balanced MBT와 Joint-KV를 결합하며, 브랜치 그룹화와 공유 캐시 재사용을 활용하는 추론 엔진으로 뒷받침된다. 적합된 스케일링 곡선은 표준 트랜스포머가 최종 모델의 검증 손실에 도달하기 위해 최대 1.26배의 학습 본체 연산량이 필요함을 나타낸다. 평가된 구성 전반에서 이 모델들은 전체 모델 KV 캐시 저장량을 최대 34% 줄이고, 단일 H200 GPU에서 최대 51.4% 더 높은 디코딩 처리량을 제공한다. 광범위한 절제 연구는 브랜칭이 폭과 깊이의 변화를 넘어서는 이득을 제공하고, 효과적인 KV 공유 전략(Joint-KV)을 가능하게 하며, 융합 함수 전반에서 견고하게 유지되고, 중간 계층에서 가장 우수한 성능을 발휘함을 보여준다. 이러한 발견은 브랜칭이 언어 모델을 더 나은 모델링 품질과 더 빠른 추론 방향으로 확장하기 위한 실용적인 경로를 제공함을 입증한다.

One-sentence Summary

Researchers at the University of Illinois Urbana-Champaign, Rice University, and colleagues propose the Multi-Branch Transformer (MBT), which processes inputs through shared initial layers, runs independently parameterized parallel Transformer blocks, and fuses outputs before shared final layers; its Balanced variant with Joint-KV shared caching outperforms sequential Transformers under matched parameter and training-compute budgets, reducing KV-cache storage by up to 34%34\%34% and increasing decoding throughput by up to 51.4%51.4\%51.4%.

Key Contributions

  • This work introduces the Multi-Branch Transformer (MBT), which processes input through shared initial layers, executes independently parameterized sequential Transformer block branches in parallel, and fuses their outputs before shared final layers; it also defines Wide-Shallow, Narrow-Deep, and Balanced branch allocation variants.
  • The paper proposes Joint-KV, which constructs a shared KV cache from parallel branch representations at each depth while retaining branch-specific queries, and combines it with an inference engine using branch grouping and shared-cache reuse to reduce full-model KV-cache storage by up to 34% and raise single-GPU H200 decoding throughput by up to 51.4%.
  • Controlled pretraining experiments on mixture-of-experts models from 0.8B to 2.1B parameters show that 2-branch Wide-Shallow MBTs outperform matched sequential Transformers under equal parameter and training-compute budgets, Balanced MBTs improve further, and fitted scaling curves indicate standard Transformers require up to 1.26 times as much training body compute to match the final models' validation loss.

Introduction

Scaling Transformer language models usually relies on depth and width choices, while recent architectures expand attention, sparse expert computation, and residual design but still depend on sequential stacks of full blocks. Prior work explores parallelism inside blocks, width-depth tradeoffs, or systems-level execution, leaving the parallel composition of full Transformer blocks underexplored. The authors introduce the Multi-Branch Transformer, which organizes full blocks into parallel branches between shared initial and final layers, and Joint-KV, which shares a key-value cache across branches at the same depth to improve long-context inference efficiency while preserving model quality.

Method

The authors introduce the Multi-Branch Transformer (MBT), which enables parallel computation at the granularity of full Transformer blocks.

An MBT processes the input through shared initial layers (the prelude), splits computation across KKK independently parameterized branches that execute concurrently, fuses their outputs, and passes the result through shared final layers (the coda). The key idea is to preserve sequential composition within each branch while allowing subsequent shared layers to build on the results of multiple transformation paths.

In comparison to related work that explores concurrency within individual blocks, allocation of width and depth, or alternative parallel computation paths, MBTs explore an orthogonal axis of parallelism at the granularity of full Transformer blocks.

Let x=(x1,…,xT)\boldsymbol{x} = (x_1, \ldots, x_T)x=(x1​,…,xT​) be an input token sequence and H∈RT×dH \in \mathbb{R}^{T \times d}H∈RT×d its hidden states at a block's input, where ddd is the residual width. Each block Tℓ\mathcal{T}_\ellTℓ​ applies causal attention followed by a mixture-of-experts (MoE) feed-forward module:

H′=H+Attnℓ(H),Tℓ(H)=H′+MoEℓ(H′).H' = H + \mathrm{Attn}_\ell(H), \qquad \mathcal{T}_\ell(H) = H' + \mathrm{MoE}_\ell(H').H′=H+Attnℓ​(H),Tℓ​(H)=H′+MoEℓ​(H′).

An MBT replaces the serial middle stack with K≥2K \geq 2K≥2 independently parameterized branch stacks Gi\mathcal{G}_iGi​. The prelude P\mathcal{P}P, fusion F\mathcal{F}F, and coda connect them:

H=P(X),Zi=Gi(H),Y^=Dec(C(F(Z1,…,ZK))).H = \mathcal{P}(X), \quad Z_i = \mathcal{G}_i(H), \quad \widehat{Y} = \mathrm{Dec}\big(\mathcal{C}(\mathcal{F}(Z_1, \dots, Z_K))\big).H=P(X),Zi​=Gi​(H),Y=Dec(C(F(Z1​,…,ZK​))).

For fusion, the authors normalize the branch outputs, concatenate them along the hidden dimension, and project the result back to the shared width ddd:

F(Z1,…,ZK)=Concat⁡(N1(Z1),…,NK(ZK))WF.\mathcal{F}(Z_1, \dots, Z_K) = \operatorname{Concat}\bigl(\mathcal{N}_1(Z_1), \dots, \mathcal{N}_K(Z_K)\bigr) W_F.F(Z1​,…,ZK​)=Concat(N1​(Z1​),…,NK​(ZK​))WF​.

To define MBTs at comparable compute, the authors propose three branch width and depth allocations.

Wide-Shallow MBTs preserve the original block width and redistribute the middle blocks across parallel branches. Narrow-Deep MBTs preserve the original sequential depth along each branch while reducing block width. Balanced MBTs interpolate between these choices by jointly adjusting branch width and depth.

To further exploit the parallel branch structure, the authors propose Joint-KV, which constructs a shared key-value (KV) cache from all branch representations at each depth while retaining separate queries for each branch, alongside an inference engine based on nano-vLLM.

In the default Independent-KV configuration, each block in each branch maintains a separate KV cache. In Joint-KV, all branches share one KV cache at each depth, constructed jointly from their pre-attention inputs. Formally, let Sˉi,ℓ∈RT×db\bar{S}_{i, \ell} \in \mathbb{R}^{T \times d_b}Sˉi,ℓ​∈RT×db​ denote the RMS-normalized input to branch iii at depth ℓ\ellℓ. The KV-cache constructions are:

KVi,ℓind=Sˉi,ℓWi,ℓind,KVℓshared=1K[Sˉ1,ℓ;… ;SˉK,ℓ]Wℓshared.\mathrm{KV}_{i, \ell}^{\mathrm{ind}} = \bar{S}_{i, \ell} W_{i, \ell}^{\mathrm{ind}}, \qquad \mathrm{KV}_{\ell}^{\mathrm{shared}} = \frac{1}{\sqrt{K}} [\bar{S}_{1, \ell}; \dots; \bar{S}_{K, \ell}] W_{\ell}^{\mathrm{shared}}.KVi,ℓind​=Sˉi,ℓ​Wi,ℓind​,KVℓshared​=K​1​[Sˉ1,ℓ​;…;SˉK,ℓ​]Wℓshared​.

At fixed branch depth and KV dimensions, the Joint-KV cache in the branch region is 1/K1/K1/K that of Independent-KV.

The inference engine employs two additional execution mechanisms. Branch grouping groups blocks at the same depth across branches into a batched execution stage. With Joint-KV, the engine first constructs the shared KV cache from the branch input representations, then computes the branch-specific attention and MoE outputs together. This reduces the number of sequential block stages. Head folding folds the branch dimension into the query-head dimension, grouping queries by their shared KV head so that all branches use a single grouped-query attention (GQA) call. This allows the attention kernel to reuse shared KV data across branches, reducing redundant memory traffic while preserving the attention function.

Experiment

The experiments pretrain standard Transformers and multi-branch Transformers across a four-scale ladder on FineWeb-Edu, systematically varying branch count, width-depth allocation, KV sharing, fusion, and branch placement. Allocating roughly one third of blocks to two parallel branches consistently improves validation loss and zero-shot downstream performance, and the benefit persists with three or four branches, especially with balanced width-depth tuning. Joint-KV preserves these quality gains while substantially reducing KV cache size and improving decoding throughput over standard Transformers. Ablations show that branching helps beyond simple widening or depth matching, and that shared KV, concat-projection fusion, and middle branch placement are the most effective choices.

Across four model scales, the selected Wide-Shallow 2-branch MBTs keep total and active-body parameters nearly identical to standard Transformers while achieving lower validation loss at every scale. The best models place a minority of physical blocks in two equal parallel branches, with equal-depth prelude and coda, and parallel block ratios cluster around one third. Loss reductions are modest, ranging from about 0.025 to 0.031. Selected MBTs reduce validation loss at every scale while keeping total and active-body parameters almost unchanged relative to standard Transformers. The best allocations consistently place roughly one third of physical blocks in two equal parallel branches, with prelude and coda depths matched in each selected configuration. Validation loss improvements are slightly larger at smaller scales, decreasing from about 0.031 to 0.025 as model scale increases.

At this matched-compute setting, multi-branch transformers with two, three, or four parallel branches consistently improve validation loss over the serial baseline. Balanced width-depth allocations, which tune block width and total depth to preserve body compute, generally reach lower loss than Wide-Shallow allocations that keep the base width and spread a fixed block budget across branches. Increasing branch count from two to four produces only modest loss changes while preserving the multi-branch advantage. Every multi-branch configuration shown improves over the serial baseline at comparable body compute. Balanced allocations achieve lower validation loss than Wide-Shallow allocations at each branch count. Balanced models use narrower blocks and more total physical blocks, while Wide-Shallow models keep the base width and reduce depth as branches increase.

At the W4 scale, Balanced MBTs with 2, 3, and 4 branches improve validation loss over the Standard model while using less KV cache per token. Joint-KV variants match their Independent-KV counterparts in loss and provide growing cache savings relative to Standard as branch count increases. The results show that shared KV across branches can retain language modeling quality while improving cache efficiency. All W4 MBT variants achieve lower validation loss than Standard while also reducing KV cache size. Joint-KV cache savings relative to Standard increase with branch count, reaching about 32% at four branches, while loss remains nearly flat across configurations.

The ablation compares branching models with sequential widening and depth-matched controls at roughly equal body compute. Wide-Shallow MBT outperforms both global and local widening baselines, while Narrow-Deep MBT outperforms Standard and shows the steepest scaling exponent. These patterns indicate that branching improves modeling quality beyond shallower or wider sequential alternatives, and that width and depth can be jointly tuned. Neither global widening nor core-only widening reproduces the gains of the branching model at matched compute. A depth-matched narrow branching model outperforms the standard sequential model and achieves the steepest scaling trend.

Across matched KV-cache comparisons at all four scales, Joint-KV consistently achieves lower validation loss than Independent-KV. The improvement is similar across scales, while Joint-KV's larger KV projection adds only a small amount of body compute. This indicates that sharing keys and values across branches is more effective than maintaining smaller private KV caches of equal total size. Joint-KV lowers validation loss at every tested scale, by 0.011 to 0.012. Joint-KV's larger KV projection adds less than 0.6 percent body compute, far too little to explain the loss improvement.

Across four model scales and matched-compute settings, multi-branch transformers with two to four parallel branches consistently improve validation loss over serial baselines while keeping total and active-body parameters nearly unchanged, and balanced width-depth allocations generally outperform wide-shallow allocations. Ablations show that these gains come from branching itself rather than from simple widening or depth scaling. Shared Joint-KV cache variants retain or slightly improve language modeling quality while reducing cache use, with savings relative to standard models growing to about 32 percent at four branches, and they consistently outperform equal-size independent KV caches at every tested scale.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp