Analysis shows Google's TPU delivers up to 50% better inference performance per dollar compared to Nvidia accelerators.

Semiconductor research firm SemiAnalysis has released the first third-party inference benchmark report for Google’s seventh-generation custom AI chip, the TPUv7 Ironwood. Under equivalent conditions, Ironwood delivers up to 50% better performance per dollar than Nvidia’s B200 and B300, signaling Google’s formal entry into direct competition with Nvidia in the AI inference chip market.

Using the Qwen3.5 397B FP8 model as its benchmark, the report compared Ironwood against Nvidia’s B200 and B300 in aggregate serving mode. At an interactive throughput of 100 tokens per second per user, Ironwood’s inference cost stands at approximately $0.181 per million tokens, undercutting the B200’s $0.222 and the B300’s $0.276 by roughly 19% and 34%, respectively.

The advantage becomes even more pronounced in low-concurrency, high-interactivity scenarios. At an interactive rate of 20 tokens per second per user, Ironwood achieves a total throughput of 9,364 tokens per second per chip, surpassing the B200’s 8,903 and the B300’s 8,925—a throughput lead of approximately 5%. When combined with lower per-chip hourly costs, Ironwood generates 50.4% more tokens per dollar than the B200 and 96.0% more than the B300.

Ironwood also maintains strong competitiveness regarding end-to-end latency. At a 20-second median response time, its cost per million tokens drops to approximately $0.098, outperforming the B200’s $0.106 and the B300’s $0.132 by roughly 8% and 25%, respectively.

However, the benchmark results are not uniformly favorable. Within a narrow band around a 30-second median response time, the B200 can still deliver superior per-dollar performance. Furthermore, Nvidia retains an advantage in FP4 precision workloads, as Ironwood currently lacks native FP4 computational support.

| Metric | Ironwood | B200 | B300 |

|---|---|---|---|

| Cost per million tokens (100 tokens/sec interactive) | $0.181 | $0.222 | $0.276 |

| Per-chip throughput (20 tokens/sec interactive) | 9,364 tokens/sec | 8,903 tokens/sec | 8,925 tokens/sec |

| Tokens per dollar advantage (vs B200/B300) | — | +50.4% | +96.0% |

| Cost per million tokens (20-sec median response) | $0.098 | $0.106 | $0.132 |

*Note: Data sourced from SemiAnalysis test report; benchmark model was Qwen3.5 397B FP8 in aggregate serving mode.*

According to Google’s internal total cost of ownership (TCO) calculations, Ironwood’s per-dollar performance advantage widens significantly under high-concurrency conditions. At 256 concurrent requests, the advantage over the B200 reaches 76.7%, and extends to 130.2% over the B300. SemiAnalysis cautions, however, that these efficiency gains incur a latency penalty: at 256 concurrency, Ironwood’s average time to first token (TTFT) rises to 5.41 seconds, compared to 3.75 seconds for the B200 and 2.40 seconds for the B300.

In non-equivalent comparison scenarios—specifically where the GB300 NVL72 leverages disaggregated serving while the TPUv7 relies on aggregate serving—the GB300 maintains a roughly 30% per-dollar performance advantage in the mid-range end-to-end latency band. SemiAnalysis projects that this gap will close once TPUv7’s disaggregated serving optimizations are fully implemented.

**Software Stack Breakthrough: TorchTPU Replaces TorchAX**

The primary bottleneck for externalizing Google’s TPUs has historically been the software stack. Previously, external developers were forced to translate PyTorch models into JAX execution via TorchAX—a workflow plagued by challenges in low-level optimization, paged attention mechanisms, and compatibility with vLLM architectures.

The new TorchTPU framework addresses this by leveraging PyTorch’s PrivateUse1 backend extension point to expose the TPU directly as a native PyTorch device. Developers can now utilize `.to("tpu")` calls while preserving familiar distributed training interfaces such as DDP, FSDP2, and DTensor. During compilation, TorchDynamo and AOTAutograd generate FX compute graphs that TorchTPU lowers to StableHLO. XLA subsequently compiles these into TPU-executable code, while underlying Pallas kernels continue to handle performance-critical operations.

This architectural shift carries significant practical implications: vLLM and SGLang can now reuse substantially more upstream model code, scheduling logic, and API structures without requiring cross-framework rebuilding between PyTorch and JAX. SemiAnalysis reports that Inferact, RadixArk, and Red Hat are actively collaborating with Google to establish TorchTPU as a first-class supported backend for both vLLM and SGLang.

Currently in private beta, TorchTPU is slated for open-source release at the upcoming PyTorch Conference in October. Following initial adaptation for the Qwen3.5 397B model, Google intends to extend support to Kimi K3, GLM5.3, and its own open-source Gemma4. SemiAnalysis anticipates that once a core set of models is optimized, the marginal cost of integrating additional models will decrease sharply, potentially positioning TPUs for Day 0 support on vLLM and SGLang.

**Kernel Optimization and Hardware Architecture**

To achieve these performance metrics, Google’s engineering team deployed extensive, targeted optimizations across TPU inference kernels. Regarding attention parallelism, the TPU backend introduced support for a hybrid configuration of 8-way attention data parallelism (DP8) and 8-way expert parallelism (EP8) to accommodate Qwen3.5’s grouped query attention (GQA) layers—which feature 32 query heads but only 2 key-value heads. This approach effectively eliminates unnecessary All-to-All communication overhead.

Communication optimizations include merging expert IDs and routing weights into a unified All-Gather operation, which reduced latency by approximately 80 microseconds per layer in DeepSeek-V3 testing—equating to roughly 4.64 milliseconds saved per forward pass across 58 layers. Furthermore, ReduceScatter collective operations were migrated to SparseCore execution and paired with double-buffering techniques to pipeline computation and communication seamlessly. Under 8k1k workloads, these adjustments yielded throughput improvements ranging from 4.1% to 14.2% across concurrency levels of 64 to 512.

In hybrid model state management, downgrading recurrent state storage precision from FP32 to BF16 halved High Bandwidth Memory (HBM) occupancy while preserving FP32 arithmetic precision within Vector Memory (VMEM). This adjustment boosted 1k8k throughput by 15% at 512 concurrency. Concurrently, optimizations to the KV cache page layout doubled available pages from 5,141 to 10,283, driving a 16.5% throughput increase in 8k1k tests at 128 concurrency and slashing median TTFT by 95%.

SemiAnalysis attributes Google’s inference cost advantage to the tight co-design of its chips, interconnect fabric, and compilers, rather than reliance on raw single-chip compute scaling. Breaking from the MegaCore architecture of the TPU v4 and v5p eras, the TPUv7 Ironwood integrates two independent compute dies linked via a high-bandwidth die-to-die interface. Each chip houses two TensorCores and four third-generation SparseCores, offers approximately six times the HBM capacity of the prior-generation Trillium, and marks the first introduction of native FP8 hardware support.

For matrix multiplication units, Ironwood utilizes a 256×256 systolic array capable of executing 65,536 multiply-accumulate operations per cycle—quadrupling the capacity of the previous 128×128 design. This architecture, however, imposes strict constraints on model tensor shapes: dimensions must be padded to multiples of 256, or Matrix Unit (MXU) utilization will degrade. For instance, in Llama 3 8B, where attention heads measure 128 dimensions, the MXU utilization ceiling for dual attention matrix multiplications on Ironwood caps at just 50%.

Regarding chip interconnect, Ironwood retains the 3D torus topology. Its fundamental building block consists of 64 chips arranged in a 4×4×4 cube, which can be scaled via optical circuit switches (OCS) into a superpod of 9,216 chips delivering 42.5 FP8 exaflops of aggregate compute. The Inter-Chip Interconnect (ICI) network operates independently of the host CPU, enabling direct activation and gradient exchange between chips and delivering near-NVLink-level bandwidth across the entire pod.

**External Customers and Competitive Landscape**

Anthropic has committed to acquiring over one million TPUs, establishing itself as Google’s largest external TPU customer, with projected usage expected to surpass DeepMind’s internal consumption by 2029. SemiAnalysis forecasts that as critical optimizations—including disaggregated prefill/decode (PD Disaggregation) and speculative decoding—are rolled out, TPUv7’s competitiveness will strengthen further, positioning it to rival Nvidia’s GB200 and GB300 NVL72 across the entire performance spectrum.

Google’s newly announced eighth-generation TPU architecture marks a strategic split between training and inference, introducing dedicated chips for each workload: the TPU 8t for training and the TPU 8i for inference. The TPUv8i (codenamed Boardfly) abandons the 3D torus topology in favor of a flattened hierarchical network built on high-radix switches. At deployment scales of 1,024 to 1,152 chips, this architecture reduces network diameter from approximately 16 hops to roughly 7 hops—a decline of over 50%—which significantly curbs tail latency in Mixture-of-Experts (MoE) routing and multi-turn agent workloads. The TPUv8i also delivers 19.2 Tb/s of ICI bandwidth and 384 MB of on-chip SRAM.