NVIDIA Transformer Engine Boosts MoE Training Efficiency for Biological Foundation Models

NVIDIA has introduced a streamlined approach to training large-scale biological foundation models by optimizing Mixture-of-Experts architectures through its BioNeMo framework and Transformer Engine. As genomic and biological models expand in parameter count and sequence length, traditional dense transformer architectures face prohibitive computational and memory costs. Mixture-of-Experts models offer a scalable alternative by activating only a subset of specialized subnetworks per token, but their efficiency has historically been limited by fragmented GPU utilization, routing communication overhead, and activation memory constraints. To resolve these bottlenecks, NVIDIA’s BioNeMo implementation introduces several engine-level optimizations. First, it replaces inefficient per-expert processing loops with Transformer Engine’s GroupedLinear primitive, which batches multiple expert matrix multiplications into a single execution call. This drastically reduces kernel launch overhead and improves GPU scheduling efficiency. Second, the framework integrates MXFP8 precision, cutting activation and weight storage in half compared to standard BF16 formats. By applying block-level scaling across thirty-two-value segments, MXFP8 maintains numerical stability while leveraging hardware-accelerated Tensor Core instructions on NVIDIA Blackwell GPUs. Third, the recipe eliminates traditional quantization and dequantization latency by fusing the gate-up projection, SwiGLU activation, routing-weight scaling, and down projection into a single GroupedMLP kernel. This fused pipeline prevents intermediate result materialization and streamlines both forward and backward passes. Benchmark tests conducted across eight NVIDIA B200 Tensor Core GPUs demonstrate the practical impact of these optimizations. The BioNeMo recipe delivers up to 2.21 times the training throughput of standard baseline implementations when scaling to the Mixtral-8x7B architecture. The framework supports configurable expert and data parallelism, allowing researchers to adjust precision to either BF16 or MXFP8 depending on hardware capabilities and memory limits. This release provides the artificial intelligence and computational biology communities with a production-ready reference for efficiently training sparse models at scale. By abstracting complex low-level optimizations into accessible recipes, NVIDIA aims to accelerate breakthroughs in genomic sequencing, protein folding, and molecular simulation. The BioNeMo implementation is now available for testing, alongside comprehensive documentation detailing the fused kernel configurations and parallel training workflows.