Analysis highlighted high-bandwidth memory (HBM) and emerging memory technologies as critical enablers for AI training and inference performance.

During a congressional hearing in the House of Representatives' Energy & Commerce Committee Subcommittee of Communication and Technology, Ronnie Vasishta, Senior VP of telecom at Nvidia, stated that mobile networks will be called upon to support a new kind of traffic—AI traffic. This AI traffic includes the delivery of AI services to the edge, or inferencing at the edge, and could reverse the general trend towards lower growth in traffic on mobile networks. Many AI-enabled applications will require mobile connectivity, including autonomous vehicles, smart glasses, generative AI services and many other applications. Supporting this creation of data from AI will require large amounts of memory, particularly very high bandwidth memory, such as HBM.

Micron announced that it is now shipping HBM4 memory to key customers for early qualification efforts. The Micron HBM4 provides up to 2.0TB/s bandwidth and 24GB capacity per 12-high die stack, utilizing the company's 1-beta DRAM node, advanced through silicon via technologies, and a highly capable built-in self-test. HBM memory, consisting of stacks of DRAM die with massively parallel interconnects to provide high bandwidth, are combined with GPUs such as those from Nvidia to allow training and inference of various AI models. The current generation of HBM memory used in current GPUs uses HBM3e memory, and at the 2025 March GTC in San Jose, Jensen Huang said that Micron HBM memory was being used in some of their GPU platforms.

The manufacturers of HBM memories are SK Hynix, Samsung and Micron, with SK Hynix and Samsung providing the majority of supply and Micron coming in third. SK hynix was the first to announce HBM memory in 2013, which was adopted as an industry standard by JEDEC that same year. Samsung followed in 2016, and in 2020 Micron said that it would create its own HBM memory. All of these companies expect to be shipping HBM4 memories in volume by sometime in 2026.

Numen, a company involved in magnetic random access memory applications, recently discussed how traditional memories used in AI applications, such as DRAM and SRAM, have limitations in power, bandwidth and storage density. The company noted that processing performance has skyrocketed by 60,000X over the past 20 years but DRAM bandwidth has improved only 100X, creating a "memory wall." Numen's AI Memory Engine is a highly configurable memory subsystem IP that enables significant improvements in power efficiency, performance, intelligence, and endurance, applicable not only to Numen's MRAM-based architecture, but also third-party MRAMs, RRAM, PCRAM, and Flash Memory. The company has developed next-generation MRAM supporting die densities up to 1GB which can deliver SRAM-class performance with up to 2.5X higher memory density in embedded applications and 100X lower standby power consumption, with solutions that are foundry-ready and production-capable today.

Coughlin Associates and Objective Analysis in their Deep Look at New Memories report predict that AI and other memory-intensive applications will drive decreased costs and increased production of emerging memory technologies such as MRAM and RRAM. These memory technologies are already available from major semiconductor foundries, scale to smaller lithographic nodes than DRAM and SRAM, and because they are non-volatile, require no refreshes and consume less power. As a result, these memories allow more memory capacity and lower power consumption in space and power constrained environments, and are also being built into industrial, enterprise and data center applications. AI will generate increased demand for memory to support training and inference while also increasing the demand for data over mobile networks, driving demand for both HBM memory and new emerging memory technologies in a future $100B memory market.