Vera Rubin NVL72 architecture achieves 67x better performance per dollar in agentic inference workloads versus competing accelerators.

Rubin represents the first platform co-designed across six integrated products for the agentic era: the Rubin GPU, Vera CPU, NVLink 6 Switch, ConnectX-9, BlueField-4, and Spectrum-6. Today, we publish the first verified agentic inference results for Rubin, measured on our proprietary benchmark, AgentX. Even on early pre-release software, these results demonstrate why such extensive co-design was essential.

At GTC 2026, Jensen Huang presented data showing the Vera Rubin NVL72 achieving 3x performance per megawatt compared to Blackwell on models ranging from O(1–3 trillion) parameters at approximately 200 tokens per second (TPS). However, real-world testing of Rubin on pre-release software reveals up to 7x better token throughput per megawatt. Jensen’s tendency to understate performance claims at GTC continues; during GTC 2024, he stated the GB200 NVL72 would deliver 30x Hopper’s performance, yet our independent testing recorded 98x better performance.

Our performance estimates indicate that even on early software builds, Rubin generates over 2x more profit per gigawatt than the Blackwell platform. As the Rubin software stack, kernel libraries, and developer expertise mature, we anticipate this profitability gap will continue to widen.

We evaluate performance using AgentX, an industry-standard agentic inference benchmark scenario. AgentX replays real-world agentic traffic across our fleet of thousands of chips, allowing inference providers and hyperscale AI labs to holistically reference the results when determining which accelerators perform best in specific scenarios. The benchmark has been widely reproduced, validated, or endorsed by nearly every major compute buyer, including Google Cloud, Microsoft Azure, Oracle, and Meta. It also enjoys support from the broader machine learning community—such as vLLM, LMCache, SGLang, PyTorch, and Hugging Face—as well as leading AI laboratories including OpenAI, MiniMax, ZAI, Qwen, and Moonshot AI.

We thank Jensen Huang, Ian Buck, Nick Comly, Kedar Potdar, Rohit Nagraj, and the Mainland China TensorRT-LLM team for their assistance in bringing up next-generation Rubin software and verifying our agentic benchmark results. If you find the open-source benchmark and data useful, please star the InferenceX GitHub repository. InferenceX is currently the only inference benchmark supporting TPUv7, NVIDIA, and AMD, with SambaNova and AWS Trainium coming soon. Given how closely the AgentX scenario mirrors real-world agentic inference workloads, AMD has committed to collaborating on the MI455X UALoE72 architecture as well.

At a high level, agentic workloads are defined by four key characteristics:

• Multi-turn sessions: Unlike traditional chatbots, these sessions involve tens or hundreds of user-assistant interactions, combining long contexts and high prefill reuse with sub-agent bursts and frequent tool calls.

• Long context accumulation: System prompts, tool definitions, and numerous turns cause context windows to expand rapidly.

• High prefix reuse: Because conversations typically progress linearly—concatenating turn n-1 output to turn n—most context is served directly from the KV cache rather than recomputed, contingent on available storage for KV tensors. As the number of turns increases, the ratio of cached to uncached input approaches one.

• Sub-agent bursts: Sessions frequently launch multiple short-lived sub-agents with fresh context, generating highly bursty KV-cache access patterns.

For a detailed breakdown of the methodology, refer to our AgentX article.

Performance per total cost of ownership (TCO) is a critical metric for evaluating AI accelerators. In the accompanying charts, the Y-axis represents total tokens generated per $1 of TCO, indicating how many tokens an inference provider can produce for every dollar spent on compute. This figure is derived by normalizing scenario throughput against all-in serving costs (cost per chip per hour multiplied by the number of chips deployed). Using the SemiAnalysis AI Cloud TCO Model, InferenceX provides multiple TCO baselines for each SKU. Default scenarios include:

• Owning at Large Hyperscaler Volume: Modeled cost per chip-hour covering hardware ownership and operations, including server and networking capital expenditures amortized over the expected useful life, plus colocation, power, and cost of capital. This reflects hyperscaler purchasing and financing terms, incorporating volume discounts and custom server configurations.

• Rent – 3-Year Commit: Market price per chip-hour for a three-year reserved instance, based on SemiAnalysis rental pricing surveys.

Users can adjust compute costs in our calculator to reflect their actual rates. Additional metrics, including on-demand rental rates and shorter-term commitments, are available in the AI Cloud TCO Model, which draws from our monthly survey of over 100 GPU customers, neocloud providers, and hyperscalers.

Vera Rubin delivers a substantial improvement over the GB300 in performance per dollar. Under apples-to-apples TRT-LLM NVFP4 dense conditions at 170 TPS, the Vera Rubin NVL72 achieves approximately 67x the total throughput per TCO compared to the GB300 Dynamo TRT-LLM configuration, assuming owning costs. At the 60–100 TPS range where most providers will actually deploy this model, Vera Rubin delivers 1.4x to 3x higher throughput per TCO than the latest GB300 TRT-LLM setup. Our test rack utilizes the production SKU featuring a 2,300W TDP and 1.5TB of CPU LPDDR5X per compute tray. (We discuss NVIDIA’s decision to halve Vera’s memory capacity in our separate accelerator and memory analysis.) This report focuses on TRT-LLM performance for Rubin, though future publications will cover vLLM and SGLang benchmarks on agentic inference workloads.

Vera Rubin achieves approximately 61% higher maximum P90 interactivity than the GB300 Dynamo TRT-LLM, reaching 276.24 TPS compared to 171.53 TPS. Notably, when utilizing the open-source SGLang stack, the GB300 can match Vera Rubin’s interactivity levels.

We acknowledge that these results stem from an early pre-release TRT-LLM software build, and performance will continue to improve, particularly at the extremes of the performance frontier (ultra-high throughput and ultra-low latency). Advancing the performance frontier and tracking iterative improvements remains the core objective of InferenceX. Additionally, Vera Rubin demonstrates significantly superior P90 end-to-end (E2E) latency at higher throughput levels. While tokens per second (TPS) often dominate performance headlines, latency metrics such as time-to-first-token (TTFT) and E2E latency are equally critical and represent de facto service-level agreement (SLA) standards in production environments.

Even when accounting for three-year rental costs—which stood at over $8.50 per chip-hour for Rubin and $5.00 per chip-hour for the Blackwell Ultra NVL72 in July 2026—the upgrade remains economically justified. At an 80 TPS P90 interactivity target, Vera Rubin generates 62% more total tokens for the same rental TCO. Across higher-interactivity segments of the performance frontier, Rubin delivers up to 16x more tokens per three-year rental TCO.

The conclusion is straightforward: acquiring or renting a Vera Rubin NVL72 is highly advantageous, as it produces significantly lower-cost tokens than competing accelerators, directly translating to higher margins. As Jensen Huang famously noted, “the more you buy, the more you earn.”

When compared to single-node Blackwell and AMD MI355X architectures, the performance-per-TCO advantage widens across the entire operational curve. At a realistic 80 TPS P90 SLA, an inference provider can serve 10x more tokens per dollar using the owning TCO model compared to the B300. This advantage extends to P90 end-to-end latency as well. Under owning cost assumptions, at a throughput of 60 million tokens per dollar of TCO, Vera Rubin achieves a P90 E2E latency of approximately 20 seconds, compared to 60 seconds for B200/B300 systems running the latest vLLM serving stack. At 160 million tokens per dollar, the latency gap expands sixfold: 20 seconds for Vera Rubin versus 120 seconds for the competition.

On agentic workloads, Vera Rubin renders the H200 virtually obsolete, drawing a direct comparison to a TI-84 calculator in modern AI infrastructure. At an 80 TPS P90 interactivity target, Rubin delivers 18x more tokens per dollar. At 120 TPS, that advantage expands to 39x. It is worth noting that Rubin’s relative advantages are less pronounced for offline batch inference and training, which remain the primary migration targets for legacy Hopper deployments among major research labs.