Google DeepMind and Google Research have unveiled Decoupled DiLoCo, a new distributed training architecture designed to make large language model training more resilient and efficient across geographically separated data centers. Traditional high-performance AI training relies on tightly coupled systems where thousands of chips must remain in near-perfect synchronization. As models grow in scale, maintaining this synchronization across vast distances becomes a significant logistical challenge, often limiting speed and increasing vulnerability to local failures. Decoupled DiLoCo, or Distributed Low-Communication, addresses these issues by dividing training runs across decoupled compute islands. Instead of requiring constant, synchronous communication, the system allows asynchronous data flow between these nodes. This design isolates local disruptions, ensuring that the rest of the system can continue learning efficiently even if one part experiences an issue. Crucially, the architecture avoids the communication delays that previously made distributed methods like Data-Parallel impractical at a global scale. The practical application of this technology was demonstrated through a successful pre-training of a 12 billion parameter model across four separate U.S. regions. The system utilized standard wide-area networking with bandwidths between 2 and 5 Gbps, eliminating the need for custom network infrastructure. Remarkably, this training run was completed more than 20 times faster than conventional synchronization methods. This performance boost is achieved by incorporating necessary communication into longer periods of computation, thereby avoiding the bottlenecks where system parts are forced to wait for one another. Beyond speed and resilience, the architecture enables the mixing of different hardware generations within a single training run. Experiments showed that chips from different generations, such as TPU v6e and TPU v5p, running at varying speeds, matched the machine learning performance of runs using uniform hardware. This capability allows organizations to leverage stranded or older resources, extending the useful life of existing infrastructure and alleviating capacity bottlenecks that occur as new hardware is rolled out gradually. By enabling training jobs at internet-scale bandwidth, Decoupled DiLoCo allows companies to tap into unused compute resources regardless of their physical location. This approach supports a full-stack evolution in AI training, where hardware, software, and research converge to optimize efficiency. The development was led by a team including Arthur Douillard, Keith Rush, and Yani Donchev, with support from senior leaders at Google DeepMind and Google Research. This innovation represents a significant step toward unlocking the next generation of AI by providing a flexible and robust foundation for training frontier models across diverse and distributed environments.