One-sentence Summary

Researchers from DreamX, Alibaba Group propose IntBMoE, a block-conditioned Mixture-of-Experts that decouples participation, execution, and materialization by pairing dense expert composition with sparse block execution through a learned codebook and Dual-Path Residual Gating, achieving gains in image classification, language modeling, and sequential recommendation, along with a 2.4% relative UVCTR lift in AMap’s deployed generative recommendation system under a 60 ms latency budget.

Key Contributions

• IntBMoE is a block-conditioned mixture-of-experts architecture that decouples expert participation, execution, and materialization by pairing dense expert composition with sparse block execution. Its blocks come from a small learned codebook, so participation is full, execution stays sparse, and materialization is bounded by the codebook size.
• Within each block, Dual-Path Residual Gating (DPRG) couples two independently composed paths through residual multiplicative gating, combining separate value and gate pathways.
• On ImageNet-1K, IntBMoE shows consistent gains over representative sparse and dense MoE baselines, and additional experiments on MiniPile and IntTravel validate generalization to language modeling and sequential recommendation. It is deployed in AMap’s generative recommendation system, serving hundreds of millions of users under a 60 ms latency budget with a 2.4% relative UVCTR gain in online A/B testing.

Introduction

Mixture-of-Experts (MoE) architectures are widely used in language, vision, multimodal, and recommendation models because they scale model capacity while controlling computation. Prior MoE designs, however, couple expert participation, execution, and materialization: sparse routing activates few experts but limits each token to selected experts; dense output mixing lets all experts contribute but requires computing every expert; parameter merging achieves full participation but creates many expert-sized parameter sets at runtime. The authors introduce IntBMoE, a block-conditioned MoE that separates reusable block construction from token-level execution. A finite codebook and shared hypernetwork compose a bounded set of reusable blocks from the full expert pool, while a router executes only a few blocks per token, achieving pool-wide expert participation, sparse execution, and bounded parameter materialization.

Method

The authors introduce IntBMoE, which integrates into a standard Transformer backbone by replacing the traditional Feed-Forward Network sublayer. Instead of utilizing standard dense or token-level expert routing, the architecture composes reusable multi-layer blocks from shared expert pools and sparsely routes each token to a select few of these blocks.

As shown in the figure below, the design operates at two distinct levels: block-level parameter synthesis and token-level routing.

At the block level, the module maintains a codebook of $K$K learned embeddings, where each embedding $\mathbf{c}_b \in \mathbb{R}^{d_c}$cb​∈Rdc​ identifies a candidate $L$L-layer block. A shared hypernetwork $h_\phi$hϕ​ processes each embedding through a linear-LayerNorm-ReLU trunk to produce a hidden representation $\mathbf{q}_b$qb​. Two linear output heads then map this representation to value and gate composition coefficients, $\boldsymbol{\alpha}_b^v$αbv​ and $\boldsymbol{\alpha}_b^g$αbg​.

For each internal layer $\ell$ℓ, the module maintains a layer-specific pool of $E$E expert bases. The hypernetwork coefficients linearly combine these expert bases to synthesize the parameters for the $K$K blocks. To maintain parameter scale stability as the pool grows, a variance-preserving factor of $1/\sqrt{E}$1/E​ is applied during composition:

$$W_{b, p}^{(\ell)} = \frac{1}{\sqrt{E}} \sum_{e=1}^{E} \alpha_{b, e}^{p} W_{e}^{(\ell)}, \quad \mathbf{b}_{b, p}^{(\ell)} = \frac{1}{\sqrt{E}} \sum_{e=1}^{E} \alpha_{b, e}^{p} \mathbf{b}_{e}^{(\ell)}$$Wb,p(ℓ)​=E​1​e=1∑E​αb,ep​We(ℓ)​,bb,p(ℓ)​=E​1​e=1∑E​αb,ep​be(ℓ)​

where $p \in \{v, g\}$p∈{v,g}. Because this synthesis depends solely on the learned block embeddings and not on token representations, the $K$K composed blocks are token-independent and can be precomputed.

At the token level, a router independently selects the Top-$k$k blocks for each token $t$t based on its representation $\mathbf{x}_t$xt​. The router, implemented as a two-layer ReLU MLP, outputs a score vector $\mathbf{r}_t$rt​, and the routing weights $\pi_{t,b}$πt,b​ for the selected blocks are computed via softmax. Before entering a selected block $b$b, the token representation undergoes block-conditioned feature filtering. The token $\mathbf{x}_t$xt​ is concatenated with the block codebook embedding $\mathbf{c}_b$cb​ and passed through a shared feature-filtering layer to generate a soft feature-wise mask $\mathbf{m}_{t,b}$mt,b​. The filtered representation $\mathbf{z}_{t,b}^{(0)} = \mathbf{x}_t \odot \mathbf{m}_{t,b}$zt,b(0)​=xt​⊙mt,b​ provides distinct views of the same token for different blocks.

Within each block, the authors employ Dual-Path Residual Gating to introduce nonlinear interactions between the independently composed value and gate paths. For an internal layer $\ell$ℓ with input $\mathbf{z}_{t,b}^{(\ell-1)}$zt,b(ℓ−1)​, the value and gate transformations are computed as:

$$\mathbf{v}_{t,b}^{(\ell)} = W_{b, v}^{(\ell)} \mathbf{z}_{t,b}^{(\ell-1)} + \mathbf{b}_{b, v}^{(\ell)}$$vt,b(ℓ)​=Wb,v(ℓ)​zt,b(ℓ−1)​+bb,v(ℓ)​ $$\mathbf{g}_{t,b}^{(\ell)} = \mathrm{RMSNorm} \left( W_{b, g}^{(\ell)} \mathbf{z}_{t,b}^{(\ell-1)} + \mathbf{b}_{b, g}^{(\ell)} \right)$$gt,b(ℓ)​=RMSNorm(Wb,g(ℓ)​zt,b(ℓ−1)​+bb,g(ℓ)​)

These paths are coupled through residual multiplicative modulation:

$$\widetilde{\mathbf{z}}_{t,b}^{(\ell)} = \mathbf{v}_{t,b}^{(\ell)} \odot \left( \mathbf{1} + \lambda \operatorname{SiLU} (\mathbf{g}_{t,b}^{(\ell)}) \right)$$zt,b(ℓ)​=vt,b(ℓ)​⊙(1+λSiLU(gt,b(ℓ)​))

where $\lambda$λ is a learnable residual scale shared across internal layers. LayerNorm is applied between consecutive layers, and the final output of the block is denoted as $F_b(\mathbf{x}_t)$Fb​(xt​).

The outputs of the selected blocks are aggregated using their respective routing probabilities to form the routed output $\mathbf{y}_t^{\mathrm{route}}$ytroute​. To capture common information that does not require token-dependent selection, the architecture adds an always-active shared SwiGLU expert $S$S. The final output is the sum of the routed block outputs and the shared expert output:

$$\mathbf{y}_t = \mathbf{y}_t^{\mathrm{route}} + S(\mathbf{x}_t)$$yt​=ytroute​+S(xt​)

Regarding computational complexity, the composition of all $K$K blocks from $E$E bases incurs a cost of $O(KED)$O(KED), while executing the $k$k selected blocks for $T$T tokens costs $O(TkD)$O(TkD). During inference, because block composition relies only on fixed model parameters, the composed block parameters can be constructed once and cached. This removes the $O(KED)$O(KED) composition term from request-time computation, ensuring that the dominant routed-block cost remains $O(TkD)$O(TkD) and is independent of the expert-pool size $E$E.

Experiment

The authors evaluate IntBMoE on ImageNet-1K with a DeiT-Tiny-style backbone and compare it against dense, sparse-routing, and dense-participation MoE baselines. IntBMoE outperforms all baselines, and ablations show that the composed gate path, two-layer block structure, shared expert, and feature filtering each contribute, with the gate path being the most important. Analyses further indicate effective full-pool expert participation, class- and depth-dependent block routing, and increasingly block-specific expert-composition recipes in deeper layers. Cached inference keeps request-time memory and computation independent of expert-pool size, and the method generalizes to language modeling and sequential recommendation, including a production POI recommendation deployment with improved UVCTR under latency constraints.

IntBMoE achieves 73.76% Top-1 and 91.48% Top-5 accuracy on ImageNet-1K, improving over its dense backbone by 7.36 and 3.79 percentage points respectively. It also exceeds all compared sparse-routing and dense-participation MoE baselines, including the strongest competitor, while using a comparable total parameter budget near 24M. Inference cost is 4.063 GFLOPs per image without caching and 3.457 GFLOPs with caching. IntBMoE outperforms the dense backbone by over seven Top-1 points and nearly four Top-5 points. It surpasses the strongest MoE baseline by close to two Top-1 points under comparable parameter budgets. Caching lowers per-image inference cost from about 4.1 GFLOPs to about 3.5 GFLOPs while preserving selective block execution.

The full model achieves the highest Top-1 and Top-5 accuracy among the ablated variants. Removing the composed gate path causes the largest performance decline, followed by collapsing the two-layer block into a parameter-matched single-layer design. Shared expert and feature filtering provide smaller but consistent gains, while fixed and softmax-normalized coefficient formulations underperform the full formulation. The composed gate path is the most critical component, with its removal causing the largest Top-1 and Top-5 drops among all ablations. The shared expert and feature filter contribute smaller but consistent accuracy gains, and learned or unconstrained coefficient formulations outperform fixed or softmax-normalized alternatives.

IntBMoE generalizes to autoregressive language modeling and sequential recommendation tasks. On MiniPile, it achieves lower perplexity than the dense backbone and all reported sparse or dense-participation baselines. On IntTravel, it attains the best ranking metrics among evaluated methods. On MiniPile, IntBMoE reduces perplexity by a clear margin over the dense backbone and by a smaller margin over the strongest baseline. On IntTravel, IntBMoE achieves the highest HR@1, HR@5, and NDCG@5 among the compared methods. The dense backbone has the weakest MiniPile language modeling metrics, while several sparse baselines improve loss and perplexity.

The full model achieves the best results on both MiniPile and IntTravel across all reported metrics. Ablations show the gate path is the most influential component for MiniPile and a major contributor for IntTravel, while the two-layer structure provides benefits beyond matched expert parameters. Shared experts, learned lambda, feature filtering, and unconstrained composition all contribute consistent gains on both autoregressive tasks. Removing the gate path causes the largest increase in MiniPile loss and perplexity and lowers IntTravel hit rates. The parameter-matched 1-layer variant underperforms the full model, showing the two-layer block structure adds benefits beyond expert parameter count. Using a shared expert yields smaller but consistent improvements across both tasks. Fixing lambda or removing feature filtering hurts both tasks, while unconstrained variance-scaled coefficients outperform softmax-normalized composition.

IntBMoE is evaluated on ImageNet-1K classification, MiniPile autoregressive language modeling, and IntTravel sequential recommendation, with additional ablation studies. It consistently outperforms its dense backbone and compared sparse or dense-participation MoE baselines under comparable parameter budgets, while caching reduces per-image inference cost from about 4.1 to 3.5 GFLOPs. Ablations show the composed gate path and two-layer block structure are most important, with shared experts, feature filtering, and unconstrained coefficient composition providing smaller but consistent gains across tasks.