SambaNova는 맞춤형 실리콘 아키텍처를 통해 메모리 벽을 극복하여 AI 추론 병목 현상을 해결합니다.

AI inference increasingly encounters memory and system-level constraints rather than a shortage of arithmetic capability. To address this, SambaNova designed its SN40L Reconfigurable Dataflow Unit (RDU) around streaming dataflow integrated with SRAM, HBM, and DDR memory. This architecture supports both large foundational models and collections of specialized models while minimizing data movement and rack requirements. SambaNova’s comparisons against Cerebras, Groq, and GPUs highlight an emerging industry shift: competition now centers on utilization, memory capacity, deployment density, and system efficiency rather than peak compute alone.

The rapid expansion of generative AI has driven accelerator designers toward higher compute targets, yet inference performance increasingly hinges on how efficiently a system delivers data to that compute. Memory bandwidth, capacity, interconnects, model switching, and software scheduling now dictate how much silicon organizations actually utilize.

SambaNova engineered the SN40L RDU specifically around these constraints. Rather than treating memory as a secondary subsystem, the design integrates streaming dataflow, compiler-controlled operator fusion, and a three-tier accelerator memory hierarchy. The company claims this architecture can execute large models using fewer processors and reduced data center footprint while maintaining BF16/FP32 precision.

SambaNova benchmarks its inference performance against Cerebras, Groq, and GPU-based services using Llama 3.1 models. For the Llama 3.1 405B model, the company reports 129 output tokens per second (TPS) per user. Notably, in SambaNova’s provided comparison, neither Cerebras nor Groq supported that specific model.

A comparison using the 70B variant reveals deeper architectural differences. SambaNova utilizes 16 SN40L chips. According to its analysis, Cerebras requires four WSE-3 wafers, totaling 336 chips, to achieve 445 TPS. SambaNova estimates that Groq relies on 576 LPUs to reach 250 TPS, noting that each LPU contains only 230 MiB of SRAM.

These figures derive from SambaNova’s internal analysis and assumptions and should not be treated as independent, apples-to-apples system benchmarks. However, they effectively illustrate the core architectural premise SambaNova presents to customers: inference efficiency is ultimately determined by the volume of hardware required to deliver useful application throughput.

Three Memory Tiers Address the Bottleneck

The SN40L’s memory architecture forms the foundation of this approach. Each socket integrates 520 MiB of distributed on-chip SRAM, 64 GiB of co-packaged HBM, and up to 1.5 TiB of directly attached DDR DRAM. The system allocates distinct workloads to each memory tier.

SRAM delivers high bandwidth directly adjacent to computational units. HBM stores actively executing models and frequently accessed data. DDR offers sufficient capacity to house larger collections of models and checkpoints near the accelerator. SambaNova notes that an SN40L node can transfer models from DDR to HBM at speeds exceeding 1 TB/s.

This configuration directly tackles a growing inference challenge. While HBM offers massive bandwidth, its limited capacity restricts how many large models or experts a processor can keep resident. Offloading weights back to host memory introduces latency and consumes PCIe bandwidth. Directly attached DDR provides a crucial intermediate capacity tier, eliminating the need to force every model into costly HBM.

Fabricated on TSMC’s 5nm process, the SN40L employs a 2.5D CoWoS chiplet package housing two Reconfigurable Dataflow Dies alongside HBM. Each socket delivers 638 BF16 TFLOPS via 1,040 Pattern Compute Units (PCUs). An additional 1,040 Pattern Memory Units (PMUs) supply distributed memory and address-generation capabilities.

The PCUs manage systolic and streaming computations, while the PMUs store tensors, parameters, metadata, and intermediate results. These resources are interconnected via a Reconfigurable Dataflow Network, which leverages programmable vector, scalar, and control fabrics.

Compiler Integration as Architecture

SambaNova shifts substantial operational responsibility into software. Its compiler translates models into dataflow graphs and distributes operations across decentralized compute and memory resources. The compiler can merge operations into large fused kernels and construct pipelines that retain intermediate data on-chip.

This strategy directly improves operational intensity—the ratio of computation performed per byte transferred through the system.

SambaNova demonstrates this impact using a Monarch FFT benchmark. Without kernel fusion, the workload achieves 39.5 operations per byte (OPB). Combining GEMM0, multiplication, and transpose operations increases this to 102.6 OPB. Full spatial fusion pushes the metric to 410.4 OPB.

Figure 1. SN40L combines dataflow compute with three-tier memory.

The primary benefit is reduced traffic to external memory. Conventional accelerators may possess vast arithmetic capability yet leave much of it idle due to data starvation. By increasing operational intensity, the architecture enables more processing to occur on data already residing within the processor.

SambaNova states that its compiler can fuse an entire Llama 3.1 8B decoder into a single dataflow kernel, allowing repeated execution without continuous kernel-launch overhead. For independent software vendors (ISVs), this shifts optimization efforts away from hand-written kernels toward compiler-directed mapping.

Composition-of-Experts Reshapes Model Hosting

SambaNova extends its architecture through a Composition-of-Experts (CoE) framework. Rather than relying solely on a single monolithic model, CoE integrates independently developed specialist models paired with a routing mechanism that selects the optimal expert based on the incoming request.

The company outlines Samba-CoE, a system comprising 150 Llama-7B experts that collectively exceed 1 trillion parameters. The architecture stores these experts in DDR, maintains the router in HBM, dynamically loads the selected expert into HBM, and executes it there.

This approach reinforces the utility of the three-tier memory hierarchy. DDR functions as a large local model repository, while HBM serves as a high-speed working set. The runtime prioritizes keeping frequently accessed models in HBM, applying a least-recently-used (LRU) eviction policy when additional capacity is required.

For enterprise AI, this capability grows increasingly vital. Organizations may deploy dozens or hundreds of specialized models instead of routing every task through a single large language model. Applications spanning coding, engineering, finance, customer service, translation, computer vision, and niche domains can each leverage tailored models while sharing underlying infrastructure.

Infrastructure Efficiency as the Primary Metric

SambaNova’s value proposition ultimately extends beyond raw tokens per second. The company urges customers to evaluate useful throughput against silicon area, rack density, memory capacity, power consumption, and deployment complexity.

This distinction carries significant weight for CIOs as inference transitions from experimental workloads to production infrastructure. A processor that tops a benchmark but demands substantially more racks, networking, memory, or power may yield a less viable deployment. Silicon architects face an identical equation: additional compute yields diminishing returns if memory and interconnects cannot sustain it.

SambaNova’s competitive claims still warrant independent validation across diverse workloads, concurrency levels, latency targets, software ecosystems, and total system power draw. Meanwhile, competitors including Groq, Cerebras, Nvidia, and other accelerator vendors continue to rapidly advance their own architectures and software stacks.

Nevertheless, the SN40L highlights a critical trajectory in AI hardware. While arithmetic throughput remains essential, application performance is increasingly dictated by memory placement, compiler optimization, model residency, interconnect efficiency, and rack-level utilization.

AI inference is shifting the industry conversation from isolated chip specifications to holistic system design. SambaNova engineered the SN40L to align with this transition. By integrating dataflow execution, a three-tier memory hierarchy, compiler-controlled fusion, and dynamic model management, the architecture treats the memory wall as a cross-cutting challenge spanning both silicon and software. Ultimately, real-world production deployments will reveal how effectively this approach reduces cost, power consumption, and infrastructure demands across varied enterprise workloads.

Editorial Assessment

SambaNova tackles a legitimate AI infrastructure bottleneck: processors require sufficient memory bandwidth and capacity to keep high-cost compute resources fully utilized. The SN40L’s SRAM-HBM-DDR hierarchy and dataflow compiler represent a cohesive solution. Its most compelling advantage may stem from deployment density rather than raw token throughput. Independent testing covering power consumption, concurrency, latency, software maturity, and total cost of ownership will ultimately define its market position.

Inflection point. The architectural emphasis on memory capacity, model residency, compiler scheduling, and system footprint signals a broader change in AI infrastructure. The industry has spent years emphasizing accelerator compute; inference makes sustained utilization increasingly important. This could mark an inflection point as enterprises deploy collections of specialized models.