SK Hynix and SanDisk publish HBF (High Bandwidth Flash) specification: 512 GiB flash cache co-packaged with GPUs to address AI inference memory wall; endorsed by Google and Tenstorrent.

HBM has already demonstrated that by stacking and co-packaging storage media close to GPUs, the data movement bottleneck in AI computation can be alleviated.

Now, SK Hynix and SanDisk have formally applied a similar approach to NAND, placing high-capacity flash storage next to various xPUs, including GPUs, through HBF to address the memory wall problem in AI inference.

Recently, the two companies released the "High Bandwidth Flash (HBF) High-Level Base Die Specification, Version 0.7.0" through OCP. Google and Tenstorrent are acknowledged in the specification as providers of feedback and recommendations.

Positioned between HBM and SSD, HBF is high-capacity flash storage situated adjacent to xPU, jointly managed by Base Die and host software.

The application scope extends from static model weights to KV Cache, multiple models, MoE, and Agent workloads.

The physical constraints of NAND in write operations, durability, and management have not been eliminated. The complexity arising from these constraints is ultimately transferred to the software stack of the Base Die and accelerators.

The memory wall problem has persisted from the CPU era into today's data-centric computing systems. When GPU performance improvements continue to outpace memory response times, data movement increasingly becomes a performance bottleneck.

Simultaneously, the growth in large language model scale and context length has created demands for higher memory bandwidth and significantly increased memory capacity.

Data involved in AI processing can be roughly divided into two categories: one includes model inputs and intermediate results generated by successive processing layers, which are dynamic and commonly referred to as activations; the other includes weights (parameters) that define the model.

During inference execution, model weights generally remain unchanged. However, their total volume may far exceed what a single processing chip or its local cache can accommodate.

For sufficiently large models, some weights may be stored on SSDs within the rack or in network-attached storage further away from the processor.

This creates a multi-tiered data path: non-volatile storage provides long-term capacity, weights needed for computation are brought into DRAM (possibly HBM), and then cached in processor-side SRAM.

Therefore, when a weight is first accessed, data may need to traverse several storage levels to reach the processor. If this weight is later evicted from cache and subsequently needs to be accessed again, the same path must be traversed once more.

The concept behind HBF is similar to the thinking that led to HBM: moving storage media closer to compute units through stacking and co-packaging.

However, the media they use and the roles they play differ. HBM provides low-latency, high read/write bandwidth working memory using DRAM; HBF attempts to leverage higher-capacity NAND to handle large-scale, read-heavy data originally stored on SSDs or network storage—such as model weights—while maintaining high aggregate read bandwidth.

In some system designs, model weights pre-loaded in HBF can bypass the traditional DRAM staging step and transfer directly to processor-side cache. While such a design complicates cache and memory management, it does save some of the transfer overhead from external storage to DRAM.

As Sharad Chole, Chief Scientist and co-founder of Expedera, describes it, HBF aims to bridge the gap between "high-bandwidth access and high storage capacity." This positioning is more precise than simply describing HBF as a faster SSD or flash-based HBM.

The OCP specification formally defines HBF as a non-coherent, memory-centric flash device situated adjacent to GPUs, TPUs, or other xPUs. Its purpose is to enhance HBM by adding terabyte-scale additional memory capacity near the processor.

Therefore, HBF neither directly replaces HBM nor simply adds capacity to it. It introduces another memory tier between high-bandwidth memory and traditional storage, keeping a larger proportion of model data physically closer to compute units.

Base Die is more than just a passive interconnect layer. While maintaining communication with the host xPU, Base Die manages the UCIe protocol and controls data movement between the host interface and NAND die.

Additionally, it is responsible for handling host and NAND commands, ECC encoding and decoding, error reporting, transfer scheduling, read/write/erase status, NAND initialization, and TSV redundancy mapping.

This division of labor creates a fundamental distinction between HBF and "simply placing flash die next to the processor": NAND stacking provides capacity, while Base Die provides the control, interface, and reliability mechanisms needed for the stack to function as a chiplet memory device.

The xPU connects to HBF via UCIe 3.0, a standard for intra-package die-to-die interconnect. On top of the UCIe (Universal Chiplet Interconnect Express) interface, AXI is used as the communication protocol. Simply put, UCIe handles the foundation of the physical and link layers, while AXI carries read/write operations between the host and HBF.

The specification provides very detailed descriptions of the UCIe protocol layer, Flit format, and AXI channels. But if the focus is only on HBF's role in the system, the key point is that it requires a dedicated chiplet interface with corresponding link layer implementations on both the xPU and HBF sides—unlike PCIe SSDs, which can be connected to existing accelerators without special modifications.

A single HBF stack can support up to 16 host channels. Each channel uses a dedicated UCIe link and provides access to its own NAND resources. Requests issued through one channel cannot access data connected to another channel.

Each channel also has its own contiguous local address space. Base Die maps this local address space to physical NAND banks, dies, and blocks. At the system level, host software is responsible for mapping the host's global address to the local address space of available HBF channels.

The result is not a single, automatically unified flash pool, but rather a channelized architecture whose performance partly depends on how the host distributes data and requests among multiple independent resources.

This distinction is important because the specification also requires that HBF and HBM be separately managed when used in the same system. HBF does not automatically become part of HBM's coherent address space. Software must decide which data belongs to which memory tier and how data should move, partition, and be accessed.

The reference configuration provided in the specification uses 16 NAND dies, 16 banks per channel, and 4 KiB NAND pages, with a total capacity of 512 GiB (where GiB is measured in binary and GB in decimal, approximately 550 GB), and allows for higher-capacity HBF stacks.

The maximum bandwidth target of approximately 3 TB/s is not because individual NAND achieves performance close to HBM, but rather through the convergence of 16 host channels combined with parallelism across multiple dies, banks, and arrays. At maximum configuration, each channel uses a 64-bit interface with data lane rates up to 32 GT/s.

To truly leverage these parallel resources, data must be distributed across sufficient channels, dies, and banks, and the host must maintain sufficient request concurrency with Base Die performing corresponding scheduling.

Therefore, approximately 3 TB/s is more accurately described as the specification target for maximum configuration, rather than measured results from actual chips or inference workloads.

Even if aggregate read bandwidth enters HBM-class levels, HBF is not equivalent to HBM. They still have significant differences in latency, write capability, access granularity, durability, and memory semantics.

HBF's core advantage is pairing high-capacity NAND with highly parallel reads, rather than achieving the same performance as DRAM across all workloads.

Model weights can range from several GiB to tens or even hundreds of GiB, must be fetched during token generation, and typically remain unchanged during inference. This makes them better suited for read-optimized, high-capacity NAND tiers compared to data that must be continuously rewritten.

As memory analyst Jim Handy summarizes the distinction: "Training continuously changes model weights, while inference typically keeps them unchanged."

However, the scope that the OCP specification assigns to HBF extends beyond static weight storage. Its application section includes single LLM serving, storing and switching multiple LLMs, mixture of experts models, multimodal models, agent workloads, AI parameter loading, and KV Cache read/write.

For a single model, the specification describes distributing model parameters across all host channels so they can be read in parallel. For multiple models, it provides two possible layouts.

Keeping multiple models in HBF can also reduce the need to reload entire models from external SSDs when the active model switches. While this doesn't eliminate model-switching latency, it can eliminate the separate SSD-to-accelerator loading step in traditional deployments.

For MoE, multimodal, and agent workloads, data placement on HBF can follow two strategies from the multiple-model case: either scatter data across all channels to pursue maximum bandwidth, or place data on designated channels prioritizing capacity allocation and workload isolation.

During the prefill stage, model layers compute and write KV Cache; after entering the decode stage, the attention module reads previously generated KV Cache and adds new cache content when generating subsequent tokens.

Therefore, unlike model weights which remain largely unchanged during inference, KV Cache continuously grows during inference and undergoes constant read/write operations.

The specification expects the host to understand the structure of LLM or AI workloads and arrange KV Cache data to optimize read/write performance. Therefore, HBF is defined not only as a parameter-loading device, but also as a possible storage target for data generated during inference runtime.

This expands HBF's potential value, especially when longer contexts and agent workloads increase memory capacity demands. But it also exposes the architecture directly to NAND's weakest characteristic—frequent writes.

The physical structure of NAND flash determines its read/write access pattern. Flash chips are internally divided into blocks, each further subdivided into pages.

The specification uses 4 KiB-sized NAND pages and supports 4 KiB-aligned burst writes. Write requests smaller than 4 KiB are not immediately written to the NAND core die but are first cached in the Base Die, with actual writing occurring only after a complete 4 KiB page is accumulated.

Within NAND blocks, HBF requires sequential writes. It does not support direct random overwrites of already-programmed pages. To rewrite any data in a block, direct overwriting is not possible; the entire block must first be erased, then written anew from the beginning in the prescribed page order.

These rules are relatively compatible with large, predetermined model weight layouts—weights are write-once, read-many data that can be written sequentially in complete pages.

But for dynamically generated data, this is challenging, as their size, lifecycle, and update patterns can change at any time during inference.

For data like KV Cache that require frequent fine-grained writes, the specification does not claim that all KV Cache workloads will perform well on HBF; actual performance will depend on how the host combines small writes, lays out KV Cache pages, and avoids patterns that lead to inefficient NAND block utilization.

Because model weights and KV Cache have different read/write patterns, the specification notes that mixing them in the same region may reduce durability and capacity utilization. Therefore, it recommends partitioning HBF at channel granularity and provides two example strategies.

Non-uniform partitioning allocates only enough channels to store active model weights and allocates remaining capacity to KV Cache.

This also indicates that HBF's data layout cannot be entirely left to hardware to complete automatically. The host must understand workload characteristics, estimate how much capacity weights and KV Cache each require, and then decide how much bandwidth and durability budget to allocate to each.

Since each HBF channel has an independent address space, host software must also map global addresses to different channels, decide how data is interleaved, and separately manage HBF and HBM.

Maintenance of the NAND medium itself is not entirely encapsulated within Base Die. Wear leveling can be performed by Base Die or controlled by the host through region remapping.

The host can adjust the mapping between logical and physical addresses to distribute programming and erase operations more evenly, but remapping commands do not automatically migrate existing data. The host must first stop relevant accesses before rewriting data to new locations.

Data retention and read disturb also require regular management. Base Die is responsible for detecting and reporting NAND status, while the host may need to refresh data, retry reads, isolate failed capacity, or wait for the device to complete recovery operations.

However, HBF does not transfer all functions handled by traditional SSD controllers to host software. Base Die remains responsible for NAND commands, ECC, request scheduling, and error detection, but data layout, partial wear management, and anomaly recovery require host participation.

Therefore, HBF's division of responsibilities differs from both HBM and traditional SSDs. The former, based on DRAM, does not face NAND's management complexity; the latter, while also using NAND, typically handles address translation, garbage collection, and wear management in the SSD controller, with upper-level software largely unaware of these processes.

HBF occupies a different state: Base Die retains device-side control capabilities but does not fully encapsulate all NAND media management. To fully leverage its capacity and parallel bandwidth, the xPU host and its software must still understand channels, data layout, and partial media status.

This is HBF's most fundamental tradeoff: it moves high-capacity NAND closer to compute, while requiring tighter coordination among hardware, firmware, runtime software, and workload data layout.

Copyright © 2011-2026 Lei Feng Net, Shenzhen Yingpeng Information Technology Co., Ltd. All rights reserved. ICP license 粤ICP备11095991号. Office phone 0755-26581864.