One-sentence Summary

Researchers from HKUST, IIIS, Tsinghua University, and the University of Edinburgh introduce Emergent MoEs (EMoE), which fine-tunes standard language models into Mixture-of-Experts architectures without introducing extra parameters by leveraging emergent modularity that arises during pre-training, achieving improved in-domain and out-of-domain generalization and scaling to Llama2-7B and Llama-30B.

Key Contributions

• Emergent MoE reframes standard pre-trained language models as implicit modular systems by splitting selected Feed-Forward Network layers into Mixture-of-Experts layers derived from emergent modularity, without adding parameters.
• Fine-tuning the emergent Mixture-of-Experts view exposes task-specific expert activation and changes parameter updates during training, while allowing the MoE split to be discarded afterward so the deployed architecture remains unchanged.
• Experiments on in-domain and out-of-domain tasks, parameter-efficient and full fine-tuning settings, and large models including Llama2-7B and Llama-30B show stronger downstream generalization than vanilla fine-tuning, robustness to configurations, and masking of neurons with negative transfer effects.

Introduction

Large language models increasingly benefit from modularity because expert-style computation can improve adaptation, efficiency, and generalization. Research shows that pre-trained transformers spontaneously develop sparse, task-sensitive neuron groups, but standard fine-tuning treats these models as monolithic, and prior modular variants mostly copy FFN layers or target inference speed rather than exploiting emergent modularity for downstream learning. The authors unlock this hidden structure by converting selected FFN layers into mixture-of-experts modules, creating Emergent MoEs, or EMoE. Their experiments show that fine-tuning EMoE improves in-domain and out-of-domain generalization, with task-specific expert routing, smoother parameter updates, and possible masking of neurons that cause negative transfer.

Method

The authors leverage the interpretation of transformer feed-forward networks as key-value memories to convert dense FFNs into sparse Mixture-of-Experts modules without adding new trainable parameters. For a standard FFN with key matrix $\mathbf{K} \in \mathbb{R}^{h \times d}$K∈Rh×d, value matrix $\mathbf{V} \in \mathbb{R}^{d \times h}$V∈Rd×h, input $\mathbf{x} \in \mathbb{R}^{h}$x∈Rh, and nonlinear activation $\sigma(\cdot)$σ(⋅), the output can be written as

Equivalently, each key column and value row can be treated as one neuron or one key-value memory pair:

This formulation makes an FFN resemble an MoE structure. In a conventional sparse MoE, the FFN is replaced by $N$N parallel experts and a gating module, producing

where $g_n$gn​ assigns an expert score and Top-$k$k activation keeps only the highest scoring experts. The proposed method externalizes this emergent modular structure by grouping existing neurons rather than initializing a new expert pool.

The expert construction is based on the observation that neurons with similar key vectors tend to be coactivated. Given a pre-trained FFN layer $\mathrm{FFN}(\cdot;\mathbf{K},\mathbf{V})$FFN(⋅;K,V), the authors partition the key vectors into $N$N expert groups using constrained clustering. Let $E_i \subset [d]$Ei​⊂[d] denote the indices of keys assigned to expert $i$i. For every $j \in E_i$j∈Ei​, the neuron $(\mathbf{K}_{:,j},\mathbf{V}_{j,:})$(K:,j​,Vj,:​) is extracted to form the $i$i-th expert $\mathrm{FFN}(\cdot;\mathbf{K}^i,\mathbf{V}^i)$FFN(⋅;Ki,Vi). Refer to the framework diagram, which shows the implicit key-value activation pattern, the clustering of keys into explicit experts, and the subsequent gating mechanism.

After experts are constructed, the method introduces a parameter-free gating module through average key computation. Instead of adding a trainable gate network, the gating matrix $\mathbf{G} \in \mathbb{R}^{h \times N}$G∈Rh×N is built by averaging the keys within each expert:

For an input $\mathbf{x}$x, the score of expert $i$i is computed as $\mathbf{x}\cdot\mathbf{G}_{:,i}$x⋅G:,i​. Because this score is proportional to the sum of pre-activation scores of keys inside the expert,

a larger gating value indicates that more neurons in that expert are expected to activate strongly. The final gating output is a binary Top-$k$k selection:

The sparse MoE output is then a weighted sum of the selected experts, using only existing FFN neurons as experts and existing key vectors as gating anchors. During downstream fine-tuning, the gate remains tied to the FFN parameters through the key-averaging rule, so the method does not introduce extra weights, extra expert initialization, or extra pre-training data.

The authors further analyze how this design affects training and inference. A key observation is that EMoE is not primarily beneficial because sparse activation changes inference directly. Instead, its main benefit comes from influencing how LoRA weights are learned. To isolate these effects, the method can be applied at different stages. In one variant, a vanilla model is first fine-tuned with LoRA and then split into EMoE experts only during inference. This setting mainly tests whether sparse activation at test time improves generalization. In another variant, the model is first transformed into EMoE, fine-tuned with LoRA under sparse expert activation, and then merged back into standard FFNs for inference. This setting tests whether EMoE improves parameter updating during training. As shown in the chart comparing training and inference variants, fine-tuning with EMoE and then merging back preserves most of the benefit, whereas applying sparse activation only at inference provides little improvement on average.

This result supports a practical training pipeline: EMoE can be used as a fine-tuning-time regularizer or expert-routing mechanism, and the resulting model can be converted back to the original dense architecture for deployment. The authors report that this property also extends beyond LoRA and can be applied in full fine-tuning settings.

The mechanistic explanation centers on negative transfer from low-scoring neurons. Since EMoE blocks some activated neurons through Top-$k$k expert selection, the authors compare expert selection rules to test whether the blocked experts are harmful. Selecting Top-$k$k experts improves over vanilla LoRA tuning, while Bottom-$k$k and Not-top-$k$k selections perform worse. This pattern indicates that the masked neurons are not simply less useful; they may actively interfere with learning. The comparison chart below summarizes the relative training accuracies under different expert-selection rules.

Experiment

Experiments evaluate EMoE with BERT, GPT2, and Llama models on in-domain and out-of-domain benchmarks, mainly under LoRA tuning, and compare it against vanilla LoRA and GMoE. The results show that EMoE consistently improves fine-tuned performance while using few or no additional trainable parameters, enhances out-of-domain generalization, and is more stable than learned gating variants. Further analyses and ablations indicate that key-vector based expert construction reveals task-dependent modularity, that EMoE primarily benefits weight updating during fine-tuning rather than inference, and that its gains come from masking negatively transferring neurons rather than from sparse activation, MoE structure, or pruning alone.

The reported BERT-Large results show that EMoE improves average in-domain performance over vanilla LoRA and the learned-gating ablation. EMoE also lowers the averaged out-of-domain rank compared with LoRA, although GMoE has the lowest reported OOD rank among the listed BERT-Large rows. Overall, EMoE provides the strongest in-domain average while maintaining improved out-of-domain ranking relative to the vanilla LoRA baseline. EMoE gives the highest reported in-domain average among the BERT-Large variants. The avg-k EMoE variant outperforms the learned-gating ablation on average in-domain performance. EMoE improves the averaged OOD rank over vanilla LoRA, while GMoE shows the lowest reported OOD rank.

Instruction-tuned Llama evaluations show that sparse expert routing improves MMLU accuracy over the single-expert LoRA baseline at both 7B and 30B scales. The 7B setting gains a small accuracy increase with modest overhead, and the larger top-k setting performs slightly better than the smaller one. The 30B setting also improves accuracy while keeping wall-clock time and FLOPS nearly unchanged. EMoE-style expert selection raises MMLU scores relative to vanilla LoRA tuning on Llama models. The 7B model benefits from sparse expert selection with only a modest increase in training cost. The 30B model achieves higher accuracy with almost unchanged computation time and FLOPS.

Across T5-base multi-task settings, EMoE configurations improve both in-distribution and out-of-distribution average accuracy over the baseline. Larger expert budgets with higher top-k values mainly increase in-distribution performance, while out-of-distribution gains are smaller and can plateau. EMoE settings outperform the baseline on both ID and OOD average accuracy. Increasing the number of experts and selected experts raises ID average accuracy, with the largest gain in the 32-expert setting. OOD accuracy improves less than ID accuracy and can saturate at a smaller expert budget.

Clustering-based expert construction improves averaged outcomes when paired with top-k selection, while random construction reduces performance relative to vanilla fine-tuning. Under bottom-k selection, both construction methods show degradation, indicating that selection strategy and expert organization jointly affect results. Clustering-based experts help under top-k selection, whereas random experts hurt performance. Bottom-k selection reduces performance for both cluster and random constructions.

Experiments evaluate EMoE on BERT-Large, instruction-tuned Llama models, and T5-base, with ablations of expert construction and selection to validate the design. EMoE consistently improves in-domain or in-distribution performance over vanilla LoRA and learned-gating ablations, while also improving out-of-domain results relative to the baseline, though some alternatives such as GMoE can be stronger on out-of-domain rank. Clustering-based experts with top-k selection are most effective, whereas bottom-k selection and random expert construction degrade performance.