Analysis examines physical, energy, and logistical limits constraining further datacenter expansion at current growth rates.

Generative AI models have not only exploded in popularity over the past two years, but they've also grown at a precipitous rate, necessitating ever larger quantities of accelerators to keep up. Short of a breakthrough in machine learning and with power becoming a limiting factor, AI's continued growth may ultimately hinge on a new kind of supercomputer, one that spans entire countries and potentially even continents. The solution is straightforward: if it's no longer practical to build bigger datacenters, start stitching together the ones you already have. Distributing large workloads across multiple machines is nothing new in high-performance computing—every modern supercomputer, AI or scientific, works this way, using high-speed interconnects such as Nvidia's InfiniBand or HPE's Slingshot to stitch together thousands of nodes.

Industry experts believe this approach is inevitable. "Distribution is inevitable," Dell'Oro analyst Sameh Boujelbene told The Register. Gilad Shainer, senior veep of networking at Nvidia, believes "in the coming generation you will see the ability to actually build those remote datacenters together and form a large, virtual, single datacenter." The infrastructure to stitch datacenters together already partially exists. High-speed datacenter interconnects (DCI) are widely employed by major cloud providers, and Nvidia and Mellanox have offered the MetroX line of products, which use dense wave division multiplexing to bridge InfiniBand compute fabrics across multiple datacenters over spans of up to 40 kilometers. The latest generation was released in late 2022, just weeks before ChatGPT sparked the AI gold rush, but was tuned more for disaster recovery and high-availability rather than large-scale AI training. According to Shainer, research is already underway to extend this range from tens of kilometers to thousands.

Extending datacenter networks across vast distances presents formidable technical challenges. AI workloads love bandwidth and hate latency—according to AMD, on average 30 percent of training time is spent waiting for the network to catch up. Light travels roughly 4.9 microseconds per kilometer through glass fibers, which means over a 1,000-kilometer span the round trip works out to nearly 10 milliseconds before accounting for protocol and processing overheads, making retransmits far more problematic. Bandwidth presents another significant obstacle: within a datacenter, scale-out networks typically feature eight 400Gbps links for an aggregate bandwidth of 3.2 Tbps, and extending this across a DCI would require multiple petabits of aggregate bandwidth. Modern optics support up to 1.6Tbps per wavelength, according to Rodney Wilson, chief technologist for research networks at optics vendor Ciena. Emerging technologies like hollow core fiber could help by reducing latencies through fewer repeaters, though it remains relatively new while dark fiber already installed in the ground uses conventional fiber.

Many of these challenges can be mitigated through software optimization, Shainer contends. Depending on how you distribute the workload across datacenters, it's possible to hide the latency while minimizing bandwidth required. For instance, if running a training workload across two physically disparate clusters, you'd distribute the workload so calculations are completed in the datacenters with only results sent across the datacenter interconnect. "The way that you run the job determines how much bandwidth you need between the datacenters," Shainer explained. "It could be 10 percent of the total [scale out network] bandwidth… It depends on how you structure the network."

For multi-datacenter training to work effectively, datacenters should ideally be homogeneous—using the same compute architecture—to avoid bottlenecks. Nvidia has been setting the stage for this through its DGX and SuperPod reference designs, which should help enable datacenters to work together seamlessly in these distributed configurations.