Groq released its 3 LPX accelerator, designed to speed up agentic AI inference workloads.

Nvidia has moved Groq 3 LPX into full production as an inference accelerator optimized for the decode phase of agentic AI. The architecture integrates 256 LP30 processors, rack-scale SRAM, and compiler-scheduled chip-to-chip communication to drive token generation at extremely small batch sizes. Artificial Analysis recorded performance exceeding 3,400 output tokens per second on Gemma 4 31B with a 100K context. Nvidia intends to pair LPX with Vera Rubin NVL72 to handle long-context, highly interactive inference workloads.

Agentic AI places unprecedented pressure on latency. Unlike traditional models that generate a single response and halt, agentic systems continuously inspect files, invoke tools, write and test code, evaluate outcomes, and repeat these sequences hundreds or thousands of times. Each iteration appends tokens to the context window, requiring subsequent inference steps to process an ever-expanding history. Consequently, token-generation latency becomes a direct bottleneck on how rapidly an agent can execute meaningful tasks.

Nvidia’s Groq 3 LPX targets this specific constraint. Following its acquisition of Groq’s technology, Nvidia has positioned LPX as an extension of the Vera Rubin platform, equipping its AI factory architecture with a processor purpose-built for interactive inference. Nebius is slated to become the first AI cloud provider to deploy Groq 3 LPX via its Token Factory production inference platform.

In benchmark testing, Artificial Analysis measured Groq 3 LPX executing Gemma 4 31B at 3,431 output tokens per second with a 100,000-token input context. At a 10K context, the median throughput reached 3,382 tokens per second. Nvidia also reported a median of 4,767 tokens per second on the coding-focused Speed-Bench workload, with 20% of tasks surpassing 5,500 tokens per second.

These figures are significant because agentic workloads prioritize individual-user responsiveness far more than conventional high-batch inference scenarios. At 3,431 tokens per second, generating 5,000 tokens requires approximately 1.5 seconds. At 100 tokens per second, the same operation takes 50 seconds. When multiplied across an agent executing dozens or hundreds of sequential reasoning and tool-use steps, decode speed fundamentally alters application behavior.

**Tensor Parallelism Meets First-Bit Latency**

Groq 3 LPX tackles this challenge through tensor parallelism (TP), which partitions computations across multiple processors and aggregates their outputs. While TP substantially increases throughput, small-batch inference introduces a communication bottleneck. The tensors exchanged between processors remain minimal, causing fixed communication overhead to dominate performance.

The underlying relationship follows this formula:

Transfer time = A + N/B

Where A denotes fixed first-bit latency, N represents the volume of data transferred, and B signifies link bandwidth. As N decreases, increasing bandwidth yields diminishing returns because A constitutes a growing proportion of the total transfer time.

This dynamic clarifies Groq’s architectural approach. Rather than relying primarily on massive link bandwidth, LPX focuses on minimizing the coordination delay that precedes each data transfer.

A single Groq 3 LPX rack houses 256 LP30 LPUs backed by 128 GB of aggregate SRAM. Each chip features 96 chip-to-chip links operating at 112 Gb/s. Crucially, the compiler maintains full visibility into compute units, memory resources, and network topology prior to execution, generating a clock-level schedule tailored to the workload.

This deterministic framework eliminates much of the real-time arbitration typically required when processors exchange data. By predetermining data paths and scheduling exact transmission windows for senders and receivers, an LPU can dispatch data precisely when its assigned clock cycle arrives, bypassing dynamic negotiation entirely.

**Compute and Communication Overlap**

Groq extends deterministic scheduling directly into the computation pipeline. The compiler orchestrates compute and communication at a granularity of 320-byte vectors. During matrix multiplication, an LPU can compute a segment of the output and immediately begin transmitting it, rather than stalling until the entire matrix operation concludes.

This overlap shrinks the communication tail that typically follows computation. It also preserves the effectiveness of TP at the small batch sizes demanded by interactive inference, where communication overhead would otherwise negate the compute gains achieved through parallel execution.

For silicon engineering teams, this highlights a deliberate architectural trade-off. Traditional accelerators allocate substantial resources to flexible execution, caching, routing, and dynamic workload adaptation. Groq sacrifices a degree of that flexibility in favor of deterministic scheduling. Because the compiler understands both the hardware and the workload beforehand, it coordinates them as a unified system.

For independent software vendors (ISVs), the critical metric shifts from isolated accelerator throughput to end-to-end time to completion. An agent that generates thousands of tokens, executes a tool, evaluates the output, and initiates another inference pass relies on consistently low latency across every operational stage.

**LPX Joins Vera Rubin**

Nvidia does not position LPX as a replacement for Vera Rubin NVL72. Available documentation outlines several complementary configurations for dividing inference workloads between the two architectures.

Figure 1. Groq 3 LPX accelerates agentic inference at scale. (Source: Nvidia)

Under standard prefill-decode disaggregation, Vera Rubin manages the prefill phase and transfers the key-value (KV) cache once per interaction. LPX subsequently handles decode operations using model weights resident in SRAM. Alternative configurations assign attention mechanisms and KV-cache management to Vera Rubin, while LPX processes feed-forward network layers. Nvidia also proposes a speculative decoding setup wherein LPX executes a smaller draft model, and Vera Rubin validates the proposed tokens.

This division of labor will likely grow in importance as AI factories transition away from treating inference as a monolithic workload. Prefill prioritizes efficient processing of extensive context, whereas decode emphasizes rapid, sequential token generation. Allocating these distinct tasks to specialized architectures enables Nvidia to optimize each processor for a narrower computational profile.

Groq 3 LPX provides Nvidia with a specialized inference architecture centered on deterministic execution, SRAM, and tightly orchestrated interprocessor communication. Its broader strategic value lies in how Nvidia integrates it with Vera Rubin. Rather than routing every stage of an agentic workload through a single processor architecture, the company can partition inference based on computational characteristics and direct each phase toward hardware engineered for that specific function.

**What Do We Think?**

Groq 3 LPX equips Nvidia with another critical architectural lever in an inference market increasingly governed by latency and agent responsiveness. Its deterministic compiler and SRAM-centric design directly resolve the communication challenges inherent in small-batch tensor parallelism. The 3,400-plus-token performance figures warrant serious attention, particularly when sustained across 100K contexts. The true validation will come from deployment economics, model ecosystem support, hardware utilization rates, and seamless integration with Vera Rubin at production scale.

**Inflection Point.** Groq 3 LPX signals that AI infrastructure may be reaching a tipping point where heterogeneous accelerators become standard within AI factories. Training, prefill, attention, and decode impose fundamentally different computational demands, a divergence that agentic AI dramatically amplifies. Nvidia’s strategy to couple Vera Rubin with LPX reflects a decisive shift toward workload-specific compute within a unified platform. If agentic workloads continue to drive inference expansion, token latency and processor specialization will likely shape system architecture as decisively as aggregate accelerator throughput.

Like it? We’ve got plenty more. Tell everyone you know. We’d love to hear from you and them.