NVIDIA Developer notes that the Vera Rubin and Blackwell platforms establish new performance-per-watt benchmarks for agentic AI workloads.

AI agents have expanded inference from single-turn interactions into multi-step workflows that reason, invoke tools, coordinate subagents, and carry growing context across turns. This shift is now visible in raw consumption: across 100 trillion tokens of real-world usage, OpenRouter’s State of AI report found that average prompt tokens per request grew roughly fourfold, while single agentic requests consume 15 times the tokens of ordinary chat. Properly characterizing hardware performance for this workload presents new challenges. A useful benchmark must capture long-context prefill, KV-cache reuse, interactive decode, tool-call gaps, and distributed mixture-of-experts (MoE) execution under realistic concurrency. It must also quantify how much of an AI factory’s power budget converts into useful agentic throughput while maintaining an acceptable user experience.

To address this, we examine SemiAnalysis AgentX, an open-source benchmark in the InferenceX suite that evaluates AI infrastructure for agentic-coding inference using replayed production-style sessions. Agentic sessions are inherently long, stateful, and variable, chaining model calls, tool use, and expanding context rather than following fixed prompt-and-response patterns. AgentX measures whether a platform can serve replayed agent traffic responsively, efficiently reuse previously processed context, and maximize agentic throughput per provisioned megawatt. Unlike legacy static sequence-length scenarios—which have been demoted to maintenance mode in InferenceX due to their limited representation of production traffic—AgentX varies concurrency to map the trade-off between throughput and interactivity. In earlier static benchmarks using DeepSeek-R1-0528 with 8K input and 1K output sequence lengths, GB300 NVL72 led H200 by up to 40x more tokens per megawatt, demonstrating strong baseline efficiency but failing to capture the dynamic nature of live agent traffic.

AgentX evaluates serving performance across prerecorded Claude Code sessions featuring interleaved reasoning and tool use. Each session is replayed turn by turn using the AIPerf client, ensuring every system receives identical recorded traffic so observed differences reflect the serving stack rather than benchmark-specific tuning. The replay preserves each session’s context and input/output sequence lengths, along with the original reasoning time and tool-call latency. These intervals maintain the timing of the original trajectory, accurately reproducing the KV-cache capacity pressure that a realistic benchmark must capture. The primary metric for AI factories is tokens per megawatt, reported against four user-experience thresholds: end-to-end normalized interactivity, standard interactivity, end-to-end latency, and time to first token (TTFT). These metrics provide clear guidance on how to best leverage each threshold for production deployment.

Recent preview results measured by NVIDIA using the SemiAnalysis AgentX workload—and pending SemiAnalysis review—highlight significant efficiency gains. On the AgentX DeepSeek V4-Pro workload, NVIDIA Vera Rubin NVL72 achieves up to 30x higher AI-factory throughput per megawatt than GB300 NVL72 at a target of 160 tokens per second per user. This demonstrates a substantial increase in agentic inference capacity while maintaining the same interactive serving target. Meanwhile, GB300 NVL72 extends its multi-generational advantage over prior architectures. Against H200 NVL8, GB300 NVL72 delivers up to 15x higher AI-factory throughput per megawatt for DeepSeek V4 Pro 1.6T on the AgentX workload, sustaining substantially more responsive agentic inference within the same power budget.

These throughput advantages translate directly into unit economics. GB300 NVL72 delivers up to 10x lower cost per million tokens than H200 NVL8, meaning operators can support materially more interactive agentic capacity under a fixed power and infrastructure budget, or deliver equivalent capacity at substantially lower operating costs. The advantage becomes even more pronounced as model scale increases. For large MoE models like Kimi K3 2.8T, GB300 NVL72 achieves roughly 80x the throughput per megawatt of H200 NVL8 at comparable interactivity. Furthermore, it extends the interactivity frontier to approximately 215 tokens per second per user, well beyond the operational range reached by H200 NVL8.

These performance gains stem from comprehensive system-level optimizations spanning the serving runtime, model kernels, and scale-up fabric. Key enablers include MoE-serving runtimes such as SGLang, TensorRT-LLM, and vLLM; DeepGEMM-based kernels; mixed-precision formats like MXFP4 and MXFP8; the NVIDIA Dynamo session-aware serving stack; and the high-bandwidth NVIDIA NVLink scale-up fabric connecting 72 GPUs for coordinated, rack-scale inference. Together, these layers allow large MoE models to sustain responsive throughput as agent sessions accumulate context, concurrency scales, and decode demand intensifies.

Vera Rubin NVL72 demonstrates what is achievable when a rack-scale system is explicitly tuned for the long-context, interactive, and distributed execution patterns of agentic inference. The broader Vera Rubin platform extends this approach across the full workflow: Rubin GPUs process large contexts and decode efficiently, Vera CPUs handle tool execution and KV-cache offload, and Groq 3 LPX unlocks ultrafast interactivity. Across the AI factory, NVLink 6, ConnectX-9, BlueField-4, and Spectrum-X move tokens, context, and tool results between resources. Software coordination is handled by Dynamo, Attention-FFN Disaggregation, NVFP4, TensorRT-LLM WideEP, and speculative decoding, which distribute execution across the most suitable processors. The objective remains straightforward: reduce recomputation and waiting, sustain interactive performance as agent sessions grow, and convert more of a fixed power budget into useful agentic output.

This analysis was made possible through the expertise and engineering contributions of Xin Li, Ankur Singh, Anthony Casagrande, Jonas Li, Po-Han Huang, Xiaoming Chen, and many other talented NVIDIA engineers.