NVIDIA’s NVFP4 Enables 4-Bit Pretraining with 16-Bit Accuracy, Boosting AI Model Efficiency and Speed

NVIDIA has introduced NVFP4, a groundbreaking 4-bit precision format designed to revolutionize large language model (LLM) pretraining by combining the accuracy of 16-bit computation with the speed and efficiency of 4-bit processing. This advancement marks a pivotal shift in how AI models are trained at scale, enabling unprecedented token throughput and infrastructure efficiency. AI workloads have surged in recent years, driven by the growing complexity and size of foundation models. Training these models demands massive compute resources, making efficiency not just a technical goal but a strategic necessity. While 4-bit precision has already proven transformative in AI inference—through NVIDIA’s earlier NVFP4 release—extending it to pretraining has been a major challenge due to the need for numerical stability, accurate gradient updates, and reliable convergence. NVFP4 for pretraining addresses these challenges with a purpose-built quantization recipe that maintains model accuracy while drastically reducing memory usage, accelerating arithmetic operations, and improving communication efficiency across distributed systems. This allows AI factories to process significantly more tokens on the same hardware, enabling faster training cycles, more experiments, and the development of larger, more capable models. The breakthrough is made possible by NVIDIA’s Blackwell architecture, the first to natively support FP4 formats. Blackwell’s GB200 and GB300 systems deliver massive FP4 FLOPs throughput, dramatically accelerating core matrix multiplication operations—essential for LLM training. Measured GEMM performance shows a 7x speedup over the previous Hopper generation, directly translating to faster forward passes, backward gradients, and overall training efficiency. To ensure stability and accuracy, the NVFP4 pretraining recipe incorporates advanced techniques such as dynamic quantization, gradient scaling, and error compensation mechanisms. These are carefully tuned to handle the volatility of gradients in low-precision training, preventing divergence and maintaining convergence. In validation experiments, a 12-billion-parameter Hybrid Mamba-Transformer model was trained from scratch using NVFP4 on a 10-trillion-token dataset. The results showed that NVFP4 achieved a validation loss curve nearly identical to that of a high-precision FP8 baseline—demonstrating stable training without the instabilities typically associated with ultra-low precision. When evaluated on downstream tasks across multiple intelligence domains, the NVFP4-trained model matched FP8 performance, proving that 4-bit pretraining can deliver production-grade accuracy at scale. This achievement unlocks new possibilities for AI development. By reducing memory and compute demands, NVFP4 enables organizations to train larger models, run more iterations, and explore more complex architectures—all while operating within fixed power and hardware budgets. It represents a paradigm shift: training smarter, not just harder. NVIDIA is collaborating with leading AI organizations including Amazon Web Services, Google Cloud, OpenAI, Cohere, Perplexity, Kimi AI, Reflection, and Runway to further refine and deploy NVFP4 in real-world applications. As AI continues to evolve, NVFP4 stands as a milestone in efficiency and scalability. It redefines what’s possible in large-scale model training, paving the way for the next generation of intelligent systems built on faster, leaner, and more sustainable foundations.