AMD acquires Taalas, a startup that hard-codes AI model weights directly into inference silicon, achieving 16K+ tokens/sec per user on demo chips.

At GTC 2026 in March, Nvidia CEO Jensen Huang revealed a critical limitation: companies seeking low-latency inference for agentic AI applications cannot achieve their goals using GPU architectures alone. To deliver the best performance across a range of latency profiles, Huang demonstrated that enterprises must split their AI inference workloads between two stages: prefill, where input tokens are processed on GPU clusters (which excel at this task), and decode, where the model generates responses—a stage that demands a massively parallel, deterministic SRAM-heavy matrix math engine like those from Grok, SambaNova Systems, and Cerebras.

The fundamental issue is simple: GPUs are excellent high-bandwidth parallel processors but deliver less-than-ideal performance on the decode phase of inference. Huang's GTC presentations showed that a system combining Nvidia's Grace CG100 CPUs and Hopper H100 GPUs could handle roughly 100 tokens per second per user before performance degraded—what he termed "medium tier" performance. The NVL72 hybrid platform, with 18 Grace CPUs and 36 Blackwell B300 GPUs, offered 3.5X better performance at 100 TPS interactivity measured in tokens per second per megawatt. An upcoming NVL72 system pairing Vera CV100 CPUs with Rubin R200 GPUs would deliver approximately 2X the TPS/MW of the Grace-Blackwell configuration, translating to 7X the performance of the Grace-Hopper system.

The Grace-Hopper setup could not exceed this interactivity threshold, though Grace-Blackwell systems could reach a "High tier" supporting 200 TPS per user at reasonable throughput, and Vera-Rubin NVL72s could achieve 3X better results. At the "Premium tier"—400 TPS interactivity—the Grace-Blackwell system's TPS/MW approached zero, whereas Vera-Rubin could deliver 10X better performance at reasonable efficiency.

These results used GPUs for both prefill and decode. But when Huang split the inference load, placing prefill on GPUs and decode on current-generation Grok LP30 accelerators, an entirely new "Ultra tier" emerged, delivering upwards of 1,000 TPS per user—likely the baseline requirement for agentic workloads. More striking: the Premium tier on Vera-Rubin NVL72 plus Grok achieved 35X the performance of Grace-Blackwell NVL72 systems without Grok acceleration, compared to just 10X improvement without specialized accelerators. GPU accelerators alone cannot sustain more than 400 TPS per user at reasonable system throughput.

Grok clusters, of course, have their own scaling limits on the decode side—you cannot indefinitely chain together millions of devices to drive arbitrarily higher interactivity—but the architectural advantage is clear. These findings explain both Nvidia's $20 billion acquihire of the Grok team in December and why AMD is pursuing its own path: partnering with Cerebras for disaggregated inference while acquiring AI inference startup Taalas this week. AMD's GPUs face identical inference-decode limitations to Nvidia's.

The Cerebras partnership benefits both companies, positioning Cerebras's anticipated fourth-generation waferscale compute engines (WSE-4) alongside AMD's Helios clusters—featuring Verano Epyc CPUs and Altair MI455X GPUs—to deliver disaggregated inference comparable to Nvidia's future Vera-Rubin-LPU combination. The challenge: AMD cannot control Cerebras technology. After substantial fundraising and a recent IPO, Cerebras commands a $50.9 billion market capitalization; acquiring it could cost AMD upwards of $60 billion despite revenues of just under $200 million per quarter—a 300X revenue multiple that carries significant risk in a market typically valued at 10X that premium.

For now, AMD has partnered with Cerebras—its only viable option as Graphcore was acquired by Softbank (Arm's owner), Nvidia effectively bought Grok, and SambaNova partnered with Intel. The company now seeks alternative approaches to inference acceleration for future roadmaps.

Taalas, the newly acquired startup, was founded by engineers from Tenstorrent and operates on an innovative principle: embedding AI inference models and their weights directly into ROM circuits, linked to massive SRAM blocks serving as on-chip KV caches. The current HC1 generation can store models with 8 billion parameters; the next-generation HC2 supports 20 billion parameters. A few tens of interconnected chips could theoretically hold inference models spanning a trillion parameters. Initial benchmarks demonstrate extremely low latencies and significantly lower per-token costs compared to Nvidia's Blackwell B200 GPUs.

Taalas's approach requires model-specific chip variants: the SRAM architecture remains constant, but the two metal layers encoding the model and weights must change for each model. However, Taalas claims that customizing an HC chip and procuring reasonable volumes—estimated at hundreds of thousands of units—costs roughly 100 times less than training a new generative AI model from scratch.

AMD has disclosed minimal details regarding its Taalas plans beyond stating it will "integrate the technology into its accelerator roadmap and develop system-level solutions with AMD Instinct GPUs." What emerges is a new era of model-specific architectures.