Next-generation AI inference chips are shifting from large SRAM to 3D-stacked DRAM, according to Technetbook.

South Korean outlet ZDNet Korea reports that next-generation AI inference accelerators are moving toward 3D stacked DRAM built on through-silicon via (TSV) packaging. The architectural shift replaces the massive on-chip SRAM used in earlier designs, which are approaching physical scaling limits in modern silicon fabrication. The hardware transition reflects broader pressure across the processor industry to meet the memory demands of massive language models.

Current-generation Groq LPU processors rely on large pools of on-chip SRAM to deliver rapid token decoding. While SRAM offers exceptional bandwidth, it creates steep scaling obstacles. Memory capacity remains constrained because advanced semiconductor manufacturing nodes yield diminishing density gains for SRAM cells. This physical limit restricts performance improvements when hosting larger neural networks across successive chip generations.

Switching to 3D stacked DRAM allows hardware designers to integrate significantly larger memory volumes vertically through TSV connections while maintaining the throughput needed for low-latency inference. The vertical packaging approach bypasses the surface-area limits that make large SRAM dies economically and technically difficult to produce at scale.

The shift mirrors similar moves among competing specialized processor makers. Cerebras, whose architecture is built around wafer-scale SRAM allocations, has previously confirmed that its upcoming CS 6 systems will vertically integrate 3D DRAM. As neural network parameter counts continue to climb, relying exclusively on on-chip SRAM is proving unsustainable for next-generation accelerator platforms.