Baidu Releases ERNIE 5.5 Flash Base Pretraining Weights

The ERNIE team at Baidu has released ERNIE-5.5-Flash-Base, a sparse mixture-of-experts language model designed to serve as a pre-training checkpoint for developers who plan to continue training models for domains, reasoning, and agent workflows. The model has 280 billion total parameters, activates 14 billion parameters per token, and was pre-trained from scratch on 33.5 trillion tokens with a training context extended to 128,000 tokens. It supports a 1 million-token context for fine-tuning and inference, and the released weights are positioned as a starting point for domain adaptation, instruction tuning, and reinforcement-learning research. The architecture emphasizes efficiency across attention, routing, residual connections, output prediction, and tokenization. ERNIE 5.5 Flash uses 43 Transformer layers with hidden size 4096 and 64 attention heads. Most layers use Heavily Compressed Attention, compressing every 128 tokens into one cached entry, while only one layer in seven retains token-level access through Multi-head Latent Attention. Sparse attention, low-rank gating, and virtual head mixing reduce compute and memory costs while preserving quality. Routing uses a Multi-Router MoE design with several sigmoid routers to improve expert selection, and the model replaces standard residual connections with Manifold-Constrained Hyper-Connections. A MultiMax output head suppresses implausible token tails while retaining several plausible alternatives, and ECHO replaces conventional multi-token prediction modules by reusing the backbone's final layer to train for future-token prediction. The new 200,000-entry tokenizer also groups digits to improve arithmetic and number-heavy text. Training relied on a curated multilingual corpus covering web text, repository-scale software, software evolution data, and long-context examples. Baidu also introduced a distributed Muon optimizer for efficient matrix-level updates across thousands of GPUs and FlashMask-4, an attention kernel supporting larger head dimensions, with reported forward-pass speedups up to 175% on SM100 GPUs relative to FlashAttention-4 MaskMod. In benchmark comparisons against similarly sized base models, including DeepSeek-V4-Flash-Base, Hy3-preview-Base, and MiMo-V2.5-Base, ERNIE-5.5-Flash-Base led or remained competitive across knowledge, reasoning, code, and long-context evaluations. It scored 93.3 on five-shot MMLU, 82.3 pass@1 on HumanEval+, and 71.5 on HELMET-RAG. The checkpoint is available on Hugging Face in FP8 and BF16 formats under the ERNIE 5.5 Flash Base Model License Agreement v1.0, with guidance for SGLang, vLLM, and fine-tuning. Baidu says a post-trained ERNIE 5.5 Flash will follow.