Amazon Web Services is quietly advancing proprietary high-speed networking architectures designed to reduce latency and bandwidth costs within large-scale AI training clusters.

Many datacenter networks are poorly engineered; Amazon’s is not. Last month, I met with AWS Vice President of Global Network Engineering Matt Rehder in a networking lab. When I made a lighthearted comment about how cage nuts can easily cut your hands, he responded with a blank stare. That reaction was unsurprising, not because the joke lacked merit, but because the manual assembly process it referenced no longer exists at AWS. Racks arrive pre-assembled, and technicians no longer screw hardware into place within their facilities. This interaction offered a glimpse behind the curtain into the highly automated infrastructure that powers cloud computing—a reality few outside the industry fully appreciate.

My colleague Thomas Claburn previously toured the facility and published a detailed analysis of AWS’s research paper on the subject. In summary, the company employs a flat, single-tier network wired in a deliberately near-random configuration. This architecture yields a significantly more resilient network that costs far less to operate, bringing readers up to speed on the technical foundation.

What received insufficient attention—and prompted my unscheduled request to tour an AWS facility—was the economic impact on customers. Their new networking approach delivers up to 40 percent greater energy efficiency. It has become the standard for most new datacenter builds, which AWS is deploying at scale, yet the technology remains largely invisible both internally and externally.

The resulting savings and cost efficiencies are substantial. So, where have those funds been allocated? Seventeen years ago, Amazon Senior Vice President James Hamilton highlighted [PDF] that traditional network vendors operated with profit margins comparable to mainframe manufacturers, whom he colorfully described as resembling "starving pigs at a trough." While the metaphor was mine, the observation held true. Hamilton’s core frustration was straightforward: "they're in my way." He and his team decided to address the issue by rebuilding the entire networking stack from the ground up using commodity hardware.

This strategic shift yielded significant results. Several years ago, AWS’s networking costs were already substantially lower than those reliant on traditional vendors. The subsequent introduction of their Resilient Network Graph architecture further slashed expenses. Given these compounding efficiencies, one must ask: where have those accumulated savings gone over recent years? I asked directly whether it was accurate to say AWS retained the margin improvements rather than passing them on to customers. Their response was unequivocal: "From a cost perspective? Effectively, yes."

While this answer might appear damning, it is essential to consider broader industry trends. Unlike many competitors, AWS has not increased prices on existing stock-keeping units (SKUs). Yes, the cost of GPU capacity blocks has risen quarterly, functioning similarly to a slower variant of Spot Instances. Additionally, AWS began charging for public IPv4 addresses a couple of years ago. Customers can still deploy the exact same 64 GB RAM instance available in 2020 at the identical price today—a figure that does not even account for inflation. Furthermore, while migrating from a Graviton4-based c8g.2xlarge to its Graviton5 equivalent, the c9g.2xlarge, incurs a 9 percent price increase, no customer is compelled to upgrade. Considering rising component costs, I find it remarkable that AWS has contained increases to just 9 percent.

Despite the granular nature of AWS billing, which features millions of SKUs, each SKU masks considerable underlying complexity. An EC2 instance is billed hourly—or metered by the second—but that single charge encompasses the CPU, memory, backplane, power consumption, physical building security, identity and access management protocols, operational personnel, and extensive networking infrastructure. One critical aspect of this infrastructure is that while inter-Availability Zone (AZ) data transfer carries a notable cost—approximately two cents per gigabyte across major regions—data movement within a single AZ remains free.

This is particularly impressive given that a single AZ often spans multiple datacenter facilities. As AZs expand and data volumes grow, AWS continues to absorb these escalating costs rather than transferring them to customers. Conversely, extracting data from AWS remains significantly more expensive. AWS’s legacy Snowball devices allowed physical data shipment to and from the cloud. Inbound transfers carried a flat fee, while outbound transfers added a per-gigabyte charge—an arrangement many found cumbersome. Over the past year, AWS introduced Interconnect-multicloud, which eliminates per-gigabyte fees entirely, charging only an hourly port rate. The offering includes a free tier granting one 500 Mbps port per provider, effectively removing AWS-side costs for traffic routed to Google Cloud Platform, Oracle, and eventually Microsoft Azure. For context, saturating that port monthly via the public internet would cost approximately $12,000.

This initiative represents an unprecedented shift in AWS’s pricing model. Although it was not the primary focus of my discussions, I requested an official statement regarding the change: "We know customers do not like rate-based network charges because it’s hard to predict their cost, which is why we are moving towards flat-rate pricing for new network products." This development warrants close attention. It is particularly noteworthy given that it emerges during a period of generally rising cloud prices rather than declining ones.

The core challenge lies in AWS’s tendency to operate invisibly. This raises a pertinent question: why am I explaining this rather than AWS itself? The company acknowledges its historical shortcomings in communicating its own achievements, despite a clear desire to do so. Traditionally, AWS has relied on customers to articulate these benefits, operating under the assumption that such advocacy would naturally accrue value to the platform. Consider the technical execution: the greatest challenge in a near-random network is not cabling, but routing. Standard protocols such as BGP, OSPF, and various proprietary systems calculate shortest paths, a metric that loses relevance when thousands of equivalent routes exist between any two points. AWS resolved this with a custom protocol called Scalable Intent-Driven Routing (SIDR, pronounced "cider"), which manages the control plane. A separate system named Spraypoint handles the actual data forwarding paths.

AWS publicly presented SIDR at re:Invent 2023 and revisited it during the "Monday Night Live" keynote in 2024. Subsequently, they published a paper on their Resilient Network Graph (RNG) architecture that omits any mention of SIDR. To my knowledge, no external analyst has linked the protocol to the underlying network. I had to verify this directly, and my assessment stands: it is not a trade secret. AWS simply failed to emphasize how critical SIDR is to the system’s functionality. Unfortunately, this communication gap is costing AWS market perception. Many in the industry incorrectly assume that neocloud providers and Nvidia reference architectures offer superior networking. They do not. It is possible to construct a neocloud with minimal networking expertise, and several companies have attempted exactly that.

I know this firsthand; I used to be among those engineers before I realized I could delegate this complexity to a cloud provider. AWS has become so adept at masking network intricacies that we rarely pause to consider how remarkable it is that near-full line-rate connectivity functions seamlessly between any two points in their environment. In datacenters built by individuals or smaller organizations, engineers must constantly manage bottlenecks, such as top-of-rack switches limiting aggregate traffic so that not every node can communicate at full speed simultaneously. I have never encountered a real-world scenario where that limitation applies within AWS. Consequently, when organizations attempt to build their own datacenters, they lack awareness of challenges that have been abstracted away for a generation, approaching the endeavor with a sense of how difficult the task truly is.