Analysis showed cloud and infrastructure providers increasingly building proprietary custom chips and systems instead of relying on off-the-shelf vendors.
The modern datacenter operates much like an appliance, especially for hyperscalers and the largest public cloud builders. Vertical integration and best-of-breed components have been opposing forces in datacenter design since mainframes first emerged six decades ago. As platforms become tightly integrated and innovation slows, companies shift back to best-of-breed solutions, swinging perpetually between the two approaches. To understand how vertical integration is evolving in public cloud, the thesis is straightforward: first, AWS aims to reinvent high performance computing (HPC) infrastructure in its public cloud, a goal shared by all top four public cloud providers; second, SmartNICs – network interface cards with compute engines for expanded processing – enable HPC at three of the top four public clouds; and third, AWS Outposts demonstrate how to correctly deploy on-premises public cloud through the "Nitro" SmartNIC, making SmartNIC upstart Pensando notably interesting in comparison.
The largest public clouds are not merely deploying SmartNICs but designing their own system-on-chip (SoC) architectures as innovation and competitive differentiation platforms. Alibaba Cloud's X-Dragon, AWS's Nitro, and Azure's Catapult are SmartNICs designed in-house at each cloud provider to offload the network software stack and management from compute resources. AWS's Nitro, now in its third generation and based on an in-house SoC designed by Annapurna Labs, offloads hypervisor functions and provides line-rate encryption and decryption by default for all data passing through – both network and local storage traffic. Nitro provides both boot and runtime hardware root of trust, presumably without using industry standard trusted platform modules (TPMs), and guest applications cannot modify Nitro or server non-volatile memory at any time. AWS Chief Technology Officer Werner Vogels called Nitro the "root of innovation" at AWS, crediting it with enabling AWS's Firecracker lightweight hypervisor, which in turn enabled serverless offerings like Lambda and Fargate, as well as AWS Outposts. Vogels described AWS high-level security this way: "Nitro is trusted, and because Nitro is trusted, then the network is trusted – but servers are never trusted." AWS reported greater than 20X performance improvement in 6 years using Nitro, measured as gross improvement in cross-sectional cluster bandwidth, with network latency improving from 12 microseconds to 7 microseconds.
Microsoft Azure's Catapult SmartNIC is now in its third generation with v3 "Dragontail Peak" mezzanine and "Longs Peak" PCI-Express boards, though Microsoft has not published detailed Catapult specifications. Azure deployed Catapult v1 ("Mount Granite") in its WCS cloud storage in 2012, Catapult v2 ("Pikes Peak" mezzanine and "Story Peak" PCI-Express boards) on all new server purchases within Bing and Azure starting in 2015, and Catapult v3 in 2017 to accelerate deep neural networks and increase network speed to 50 Gb/sec in Bing. Catapult v3 appears to use a Mellanox ConnectX-3 Pro chip alongside Intel's Arria 10 FPGA, with ConnectX-3 Pro chips having first appeared in OCP network mezzanine designs in 2014. Microsoft's BrainWave beta deep learning service forked from Catapult v3 to enable third-party FPGA services for deep neural networks, deploying this capability to production in December 2019 as the Azure PBS instance type, still based on Intel Arria 10 FPGAs. Azure positions its FPGA choice as a point on the migration path to custom-designed ASICs, expecting to move to custom-designed logic when cloud needs become stable enough for a four-to-five-year useful lifetime without radical reprogramming.
Alibaba Cloud's X-Dragon SmartNIC is now on its second generation (X-Dragon II), following announcement of the first generation in 2017. The second-generation chip enabled Alibaba's lightweight in-house Dragonfly hypervisor, similar in spirit to Firecracker, along with SR-IOV and the ability to live migrate applications. Because of X-Dragon II, Alibaba Cloud claims its unexpected server failure rate has decreased by a factor of 10.