SambaNova + Nvidia H200 platform benchmarked at 763 tokens/second on MiniMax inference; demonstrates heterogeneous compute viability.

Intel's bet on SambaNova appears to be paying off. This week, the AI chip startup shared benchmark results showing its latest generation of AI acceleration, which combines Nvidia GPUs with its own accelerators and substantially outperforms GPU-only inference platforms.

Testing by Artificial Analysis found SambaNova's SN50-series accelerators, announced in February, achieving 763 tokens per second in MiniMax M2.7 at short context lengths (10,000 input tokens)—several times faster than competing GPU-only inference providers. For longer context lengths, the platform sustains more than 450 tokens per second.

This performance came from combining four Nvidia H200 GPUs with 16 SN50 Reconfigurable Dataflow Units (RDUs) in a heterogeneous compute platform. The computationally intensive prefill phase—where prompts are processed and key-value caches are generated—ran on the H200s, while the memory-bandwidth-bound decode phase ran on the 16 SN50 accelerators.

Disaggregating prefill from decode has become essential for reducing token costs in long-running AI applications like code assistants. Nvidia pioneered this approach with its NVL72 rack systems, varying the ratio of GPUs for prefill versus decode, and later extended it with Groq-based LPX racks revealed at GTC this spring. Since then, AMD, AWS, Cerebras, and others have announced disaggregated or heterogeneous inference platforms.

SambaNova aims to show customers how they can extend the life of aging GPU fleets using its systems as decode accelerators. A key advantage: its air-cooled systems can be deployed in existing datacenters, unlike Nvidia's latest Rubin GPUs, which require liquid cooling.

The company plans to demonstrate even more powerful configurations using 128 and eventually 256 accelerators to show its ability to maintain high token generation rates at high throughput—something GPUs alone have historically struggled with and a key factor behind Nvidia's Groq acquisition last year.

These results arrive a month after SambaNova and Intel announced that Vector Core Compute would be among the first to deploy the GPU + RDU offering, with TogetherAI as their initial large-scale customer.

Production ramps for new chips are capital-intensive, but funding shouldn't be an issue for SambaNova's fifth-generation part. On Wednesday, the company completed the first close of a $1 billion Series F funding round led by General Atlantic, valuing the AI chip startup at $11 billion.