Micron delivers high-capacity HBM memory optimized for generative AI workloads, addressing memory bandwidth bottleneck.

Micron Technology, Inc. has begun sampling the industry's first 8-high 24GB HBM3 Gen2 memory, featuring bandwidth greater than 1.2TB/s and pin speed over 9.2Gb/s—representing up to a 50% improvement over currently shipping HBM3 solutions. With a 2.5 times performance per watt improvement over previous generations, Micron's HBM3 Gen2 offering sets new records for critical artificial intelligence data center metrics of performance, capacity and power efficiency. The company plans to reduce training times of large language models like GPT-4 and beyond while delivering efficient infrastructure use for AI inference and superior total cost of ownership.

The foundation of Micron's solution rests on its industry-leading 1β DRAM process node, which allows a 24Gb DRAM die to be assembled into an 8-high cube within an industry-standard package dimension. A 12-high stack with 36GB capacity will begin sampling in the first quarter of calendar 2024, providing 50% more capacity for a given stack height compared to existing competitive solutions. The improved power efficiency stems from Micron advancements including doubling of through-silicon vias over competitive HBM3 offerings, thermal impedance reduction through a five-time increase in metal density, and an energy-efficient data path design.

Micron's HBM3 Gen2 addresses increasing demands in generative AI for multimodal, multitrillion-parameter AI models. With 24GB of capacity per cube and more than 9.2Gb/s of pin speed, training time for large language models is reduced by more than 30%, resulting in lower total cost of ownership and unlocking a significant increase in queries per day. For an installation of 10 million GPUs, every five watts of power savings per HBM cube is estimated to save operational expenses of up to $550 million over five years. According to Praveen Vaidyanathan, vice president and general manager of Micron's Compute Products Group, the product features "a fully programmable Memory Built-In Self Test that can run at the full specification pin speed," positioning the company "for improved testing capability with our customers, creates more efficient collaboration and delivers a faster time to market."

Micron is a partner in TSMC's 3DFabric Alliance, with TSMC having received samples of the HBM3 Gen2 memory for further evaluation and tests. Ian Buck, vice president of Hyperscale and HPC Computing at NVIDIA, stated: "At the core of generative AI is accelerated computing, which benefits from HBM high bandwidth with energy efficiency. We have a long history of collaborating with Micron across a wide range of products and are eager to be working with them on HBM3 Gen2 to supercharge AI innovation."

Micron developed this product by leveraging its global engineering organization, with design and process development in the United States, memory fabrication in Japan and advanced packaging in Taiwan. The company previously announced its 1α 24Gb monolithic DRAM die-based 96GB DDR5 modules and has introduced the 1β 24Gb die-based 24GB HBM3 offering. In the first half of calendar 2024, Micron plans to make available its 1β 32Gb monolithic DRAM die-based 128GB DDR5 modules, demonstrating the company's leading-edge technology innovations for AI servers.