Nvidia announces that its Groq 3 LPX accelerator achieves 30 times higher throughput per megawatt compared to prior generations during mass production ramp.

Nvidia announced on Monday, August 24 (U.S. Eastern Time), that its Groq 3 LPX rack-scale system for interactive AI inference has officially entered full mass production. This milestone marks the commercial deployment of the company’s low-latency AI inference technology, following its approximately $20 billion (roughly NT$640 billion) acquisition of AI chip startup Groq’s related assets last year. The first systems will be deployed at AI cloud computing service provider NEBIUS (NBIS), with plans to go live later this year.

Nvidia Senior Director Dion Harris told media that the Groq 3 LPX will operate alongside Vera CPUs and Rubin GPUs within NEBIUS data centers. He noted that for cloud providers offering AI inference services, reduced latency enables them to charge premium rates to customers demanding faster response times.

On the performance front, Nvidia cited independent benchmark data from Artificial Analysis showing that Groq 3 LPX achieved 3,400 output tokens per second when running Google’s Gemma 4 31B model with a 100,000-token context window—a record for that model. For low-latency workloads such as agentic programming, response speeds reached up to four times that of the nearest competing platform.

Nvidia also unveiled new test results for the Vera Rubin NVL72 under real-world agentic workloads. Using SemiAnalysis’s AgentX workload paired with the DeepSeek V4 Pro model, Vera Rubin delivered up to 30 times the throughput per megawatt compared to the previous-generation GB300 NVL72, while reducing cost per token by up to 35 times.

The company emphasized that agentic tasks differ fundamentally from traditional chat or summarization. Task context accumulates continuously across hundreds of steps, potentially reaching hundreds of thousands of tokens. Citing OpenRouter data, Nvidia noted that agentic AI workloads can consume up to 15 times more tokens than simple chat requests. An agent might first query a database, then search news and documents, call sub-agents for analysis, execute code, and verify results repeatedly—all while context accumulates throughout the process.

This dynamic has introduced a new metric for AI infrastructure economics: how many effective agentic tasks can be completed per megawatt of electricity? As power and data center resources increasingly become bottlenecks for AI expansion, maximizing tokens produced per unit of electricity may prove more critical than simply maximizing peak compute.

From an architectural standpoint, Groq 3 LPX is not a standalone product but part of Nvidia’s “extreme co-design” approach for Vera Rubin. The entire system is engineered around compute, networking, storage, and software, encompassing Rubin GPUs, Vera CPUs, Groq 3 LPX, BlueField-4 DPUs, Spectrum-6 SPX, and other components.

“This isn’t about replacing GPUs,” Harris stated. “It’s about using the most appropriate processor in terms of price and performance for different parts of the workload.”

A defining feature of the Groq chip architecture is the integration of 500 MB of high-speed SRAM directly on-chip, designed to eliminate data transfer bottlenecks inherent in traditional memory access. Unlike conventional data center GPUs that rely on high-speed DRAM such as GDDR7 or HBM4, Groq chips depend entirely on on-chip SRAM, which operates several orders of magnitude faster than the best HBM stacks available today. The third-generation Groq chip, launched as part of the Vera Rubin platform, claims 150 TB/s of memory bandwidth.

However, SRAM consumes significant die area, limiting easy capacity scaling within a single chip. Nvidia’s highest-spec Rubin GPU features 288 GB of on-chip memory, whereas each Groq 3 LPU holds only 500 MB—a 576-fold difference. This is insufficient to run the Gemma 4 31B model on a single LPU, so Nvidia utilizes Ethernet to distribute the model across multiple accelerators. Each LPX rack accommodates up to 256 LPUs, providing a combined 128 GB of high-speed SRAM. Nvidia currently integrates 256 Groq 3 chips into a single LPX rack. While Nvidia’s primary GPUs are manufactured by Taiwan Semiconductor Manufacturing Company (TSMC), the Groq chips are fabricated by Samsung.

It is worth noting that Gemma 4 31B represents a favorable scenario for this hardware. The model is dense, meaning all 31 billion parameters are activated for every generated token. This figure aligns closely with the active parameter count of large mixture-of-experts (MoE) models like DeepSeek V3. However, running a 671B-parameter MoE model such as DeepSeek V3 would require 1,342 accelerators—more than five LPX racks. Additionally, MoE models incur performance overhead absent in dense models, as different parameters are routed for each token.

On the same day, Nvidia announced that SpaceXAI will adopt Vera CPUs to accelerate next-generation agentic AI applications and expand its Vera Rubin-based platform buildout as compute capacity scales toward multiple gigawatts. Furthermore, SpaceXAI plans to deploy an optimized version of the Vera Rubin NVL72 in space for its first-generation Starmind AI satellites.

Vera is specifically engineered for CPU workloads beyond model inference, including tool calling, code execution, data processing, task orchestration, and simulation. Featuring 88 Nvidia-designed Olympus cores and equipped with high-bandwidth LPDDR5X memory delivering up to 1.2 TB/s, Nvidia claims Vera completes tasks up to 1.8 times faster than x86 CPUs in agentic AI, reinforcement learning, and data processing workloads.

For SpaceXAI, Vera’s value extends beyond raw CPU performance. During agent execution, a substantial portion of tasks occur between GPU inference steps—executing code, processing data, calling tools, and coordinating agents. If CPUs cannot handle these tasks swiftly, GPUs may sit idle, reducing overall AI factory utilization.

As the industry shifts from large-scale model training to rapidly growing inference demand, competition in the low-latency inference market is intensifying. Earlier this year, AMD announced it would integrate its rack-scale AI systems with Cerebras chips, also targeting low-latency AI inference.

According to Artificial Analysis leaderboards, under identical Gemma 4 31B, 100K context conditions, Cerebras delivers 882 tokens per second, while Nvidia’s Groq 3 LPX achieves 3,400 tokens per second—a clear 4x gap. Market analysts, however, note important nuances. Cerebras typically runs models at mixed precision, and the entire Gemma 4 31B model can fit into one or two 44 GB CS-3 accelerators, whereas Nvidia requires at least 64 chips. Moreover, Cerebras recently announced its next-generation CS-4 accelerator, built on the WSE-3T architecture. It doubles compute, I/O bandwidth, network speed, and memory bandwidth compared to existing chips while tripling the number of accelerators per rack. The new system is expected to launch on the Cerebras Inference Cloud later this year, paired with a heterogeneous GPU+WSE setup similar to Nvidia’s NVL72+LPX configuration.

OpenAI’s recently announced Ultrafast mode promises 750 tokens per second, with Cerebras providing the underlying compute. However, test conditions and application scenarios vary across systems, and these figures should not be treated as direct performance comparisons.

Nvidia CEO Jensen Huang projected in March, upon unveiling Vera Rubin and Groq 3 LPX, that cumulative sales from the current Blackwell chips through the next-generation Vera Rubin systems could reach $1 trillion by 2027. He also revealed at the time that...