High-Bandwidth Memory (HBM) demonstrates significant performance gains for memory-intensive AI workloads.
High-Bandwidth Memory (HBM) is emerging as a disruptive technology in enterprise data centers, offering significantly faster performance than incumbent memory chip technologies while using less power and requiring less space. It is becoming particularly popular for resource-intensive applications such as high-performance computing (HPC) and artificial intelligence (AI). However, mainstream adoption for routine business applications remains distant due to HBM's cost, potential heat management challenges, and possible requirements for application rewrites. HBM was created by US chipmaker AMD and South Korean supplier SK Hynix, with development beginning in 2008. In 2013, the companies handed the specification to JEDEC, the standards body for the semiconductor industry. The HBM2 standard was approved in 2016, and HBM3 was officially announced in January. The primary manufacturers of HBM memory chips today are South Korea's Samsung and SK Hynix, along with Micron Technology.
HBM was designed to address the lagging performance and power efficiency of standard DRAM compared to CPU and GPU performance. The key innovation lies in the bus architecture: while the standard DRAM bus is 4- to 32-bits wide, the HBM bus is 1,024-bits wide—up to 128 times wider according to Joe Macri, corporate vice president and product CTO at AMD and co-developer of HBM memory. HBM chips are drastically smaller than GDDR memory, which it was designed to replace: 1GB of GDDR memory takes up 672 square millimeters versus just 35 square millimeters for 1GB of HBM. Rather than spreading out transistors horizontally, HBM stacks them up to 12 layers high and connects them using through silicon via (TSV) technology, which runs through the layers like "an elevator runs through a building," greatly reducing data travel time. With HBM sitting directly next to the CPU or GPU, less power is required to move data, and the CPU and HBM communicate directly, eliminating the need for DIMM sticks. As Macri explained, "The whole idea that [we] had was instead of going very narrow and very fast, go very wide and very slow."
Current vendor adoption shows HBM's use in high-performance systems. Fujitsu was the first to use HBM for HPC with its Arm-based A64FX processor; the Fugaku supercomputer powered by the A64FX debuted at the top of the Top 500 supercomputer list in 2020 and has remained there since. Nvidia is using HBM3 on its forthcoming Hopper GPU, while AMD uses HBM2E on its Instinct MI250X accelerator, and Intel plans to use HBM on some Sapphire Rapids generation Xeon server processors as well as the Ponte Vecchio GPU accelerator for the enterprise. Paresh Kharya, senior director of product management for accelerated computing at Nvidia, notes that standard DRAM is not well suited for HPC use and that while DDR memory can come close to HBM performance, "you'll have to have a lot of DIMMs, and it's not going to be optimal" in terms of energy efficiency.
Two major barriers threaten mainstream adoption. First is cost: according to Macri, the price difference between HBM and DDR5 at the same capacity exceeds 2 to 1—meaning 1GB of HBM costs twice as much as 1GB of DDR5. Daniel Newman, principal analyst with Futurum Research, notes this creates "a chicken and egg thing there that if it's costly to build, then it's not going to be widely used in a broad market. And so that's going to reduce the volumes that ship." Second is heat management. With five or more memory chips sharing the same cooler, Newman warns: "That means that the processor is dissipating gobs of power, all in a little tiny package, so you're going to have a heat problem. Every processor that uses HBM has to have extraordinary heat management." Macri counters that performance is what improves total cost of ownership, arguing that "if you get double performance, you improve the TCO by double," though this assumes costs remain flat.