At Hot Chips 2026, Nvidia presented the Groq 3 LPX architecture and published the first third-party benchmark for the LP30-based rack, which is already in production.

At Hot Chips 2026, Igor Arsovski, Nvidia’s vice president of hardware and former chief architect at Groq, unveiled the architecture of the Groq 3 LPX inference rack and published the first third-party benchmark of the silicon. Artificial Analysis measured the system at 3,431 output tokens per second on a 100K-context Gemma 4 31B reasoning workload, roughly four times the 870 tokens per second recorded by the next-fastest public endpoint. Arsovski confirmed the rack is already in production, built around the LP30 chip Nvidia secured through its $20 billion Groq acquisition in December 2025—a deal that simultaneously removed the Rubin CPX from Nvidia’s roadmap.

The Artificial Analysis test ran on a private, pre-release Gemma 4 31B endpoint served through Google Cloud, calculating the median of 50 sequential client requests at a concurrency of one. By contrast, the public providers evaluated against operated shared production serverless endpoints. Serving a single request at a time yields the highest per-user token throughput the hardware can achieve, though it is not directly comparable to the multi-tenant conditions governing other endpoints. During his presentation, Arsovski also showcased a higher Nvidia-internal figure of 10,996 tokens per second on the same 31B model, which he labeled “self-reported” before emphasizing the company’s goal of securing “third-party verified independent benchmarks that you guys can trust.” He noted that Gemma 4 31B is a dense model small enough to fit within a single LPX rack, leaving the trillion-parameter mixture-of-experts scale—where memory capacity becomes the primary constraint—unaddressed.

Each LP30 chip carries approximately 500MB of on-die SRAM and eliminates HBM entirely. A full LPX rack comprising 256 chips delivers 128GB of memory, 40 PB/s of aggregate bandwidth, and 315 PFLOPS of FP8 compute, with 350 nanoseconds of chip-to-chip latency. The system operates within a Vera Rubin-compatible, MGX liquid-cooled chassis that scales beyond 1,000 LPUs. By keeping model weights resident in SRAM rather than streaming them from HBM, the architecture removes the memory-access latency that typically dominates single-token decoding. It achieves this by discarding caches, branch prediction, and out-of-order execution in favor of a fully deterministic pipeline scheduled by the compiler at clock-cycle granularity. The design traces directly back to the Tensor Streaming Processor originally described by Groq—founded by ex-Google TPU engineer Jonathan Ross—in a 2020 ISCA paper titled *Think Fast*, a title Arsovski and Raghavan reused for their Hot Chips presentation.

For context, a Rubin GPU houses 288GB of HBM4, roughly 576 times the memory of a single LP30. Consequently, a 31-billion-parameter model at FP8 requires approximately 62 LPUs to hold its weights, while large mixture-of-experts models demand four-figure chip counts across multiple racks. This capacity trade-off underscores why Nvidia positions the LPU specifically for decoding rather than as a general-purpose GPU replacement. Deterministic execution also enables the compiler to predict power draw cycle by cycle, allowing Nvidia to pre-order current from the rack’s regulators ahead of demand. This approach cuts voltage droop by more than 60% and overshoot by more than 70% compared to uncompensated loads. Similarly, per-block scheduling equalizes heat distribution across the chip rather than throttling to the hottest tile, yielding roughly 10% to 11% additional performance under a fixed thermal limit. “By doing this, we can actually get more utilization of the chip under the same thermal limit, basically,” Arsovski explained. “So we can actually get, again, about 10 to 11% more performance under the same thermal limit. So this is another benefit of deterministic execution.”

Across racks, Nvidia synchronizes chips to a single virtual clock using what it terms a plesiosynchronous network. Each chip functions as both processor and router, eliminating the need for adaptive routing or congestion sensing, while clock drift is compensated directly at the chip-to-chip links. When asked during Q&A about the impact of a mid-workload chip failure, Arsovski stated that users “would experience the exact same as any other hardware in the industry” and would “just checkpoint it or reconfigure the hardware.”

Nvidia is marketing the LPX rack as a decode co-processor integrated alongside the Vera Rubin NVL72, where Rubin GPUs manage the compute-intensive prefill phase and KV cache construction while the LPUs generate output tokens. The company demonstrated three workload partitioning strategies: disaggregated prefill and decode; attention-FFN disaggregation, which retains attention and its cache on GPU HBM while the LPU executes feed-forward layers; and external-draft speculative decoding, where a compact model on the LPU proposes tokens that the GPU verifies in parallel, with only draft tokens traversing the interconnect. An FPGA bridges the synchronous LPU domain with the asynchronous environment of host I/O and GPU handoffs, orchestrated by Nvidia’s Dynamo runtime and an LPU extension to CUDA. Nvidia reports these configurations deliver three-to-five-times gains over Rubin alone on a two-trillion-parameter workload featuring a 400K-token cached context, figures measured internally by the company.

In the same Hot Chips session, Cerebras presented its CS4 wafer-scale system. Chief system architect Jean-Philippe Fricker reported the platform runs up to 30 times faster than GPUs, doubles the token rate of its CS3 predecessor, and offers ten times the token capacity. Each CS4 rack integrates three wafer-scale engines into a modular Nexus platform built around pluggable compute “backpacks” that isolate power, compute, and I/O. Fricker specified its memory bandwidth at 43 PB/s, noting it was “2,000 times higher memory bandwidth than Nvidia's next-generation Rubin chip.” Cerebras has also partnered for the prefill workload, agreeing in July to pair AMD Helios GPUs for prefill with its wafer-scale engines for decode—a division of labor that mirrors Nvidia’s in-house Groq integration.

Strategically, Nvidia withdrew its Rubin CPX, a GDDR7-based long-context accelerator, to prioritize shipping the LPU this year, a move outlined by VP Ian Buck at GTC 2026. The $20 billion Groq transaction was structured as a non-exclusive IP license combined with the recruitment of Ross, president Sunny Madra, and the majority of Groq’s engineering staff, a configuration designed to bypass formal merger review. Arsovski opened his presentation by calling it “a pinch me moment for the Groq team that's now integrated into the Nvidia group.” Meanwhile, regulatory scrutiny continues: Senators Elizabeth Warren and Richard Blumenthal wrote to the FTC and Nvidia in early 2026, contending the arrangement effectively acquired Groq “in all but name.” As of late August, no formal, deal-specific investigation has been confirmed.