Microsoft Unveils BitNet: A 2-Billion-Parameter Open-Source Model with 1.58-Bit Precision for Efficient Training

Microsoft has recently released a new open-source large language model called BitNet b1.582B4T. This model boasts 20 billion parameters and is uniquely trained using a 1.58-bit low-precision architecture, setting it apart from traditional post-training quantization methods. The result is a significant reduction in the computational resources needed for training. Microsoft reports that BitNet’s non-embedding memory footprint is just 0.4GB, far lower than other similar models in the market. For instance, Gemma-31B has a 1.4GB footprint, and MiniCPM2B requires 4.8GB. This efficiency not only enhances the model’s performance but also opens up new possibilities for deployment on resource-constrained devices and in scenarios with limited computing power. BitNet’s release introduces fresh ideas and tools to the field of large language models, potentially paving the way for more resource-efficient training methods and applications. This technological advancement underscores Microsoft’s commitment to innovation and leadership in artificial intelligence. By making BitNet open-source, Microsoft is inviting developers and researchers worldwide to contribute to and improve the model. This collaborative approach lays a strong foundation for future iterations and helps expand the scope of AI applications.