Google's TPU delivers 57% faster inference on DeepSeek's Kimi model compared to Nvidia GPUs while using DeepSeek's open-source vLLM framework.

Sixteen Google TPU v7s running Kimi K3 achieved 709 tokens per second—57% faster than Nvidia's GB200 at 452 tokens per second.

The company behind this result is Inferact, a newly founded inference startup and not, as one might expect, either Moon-Whispering Intelligence (creators of Kimi) or Google (makers of TPU). Inferact's founding team consists of the original vLLM developers, who raised a $150 million Series A led by a16z and Lightspeed (with participation from Zhenfund, Sequoia, and Altimeter) at an $800 million valuation.

The breakthrough rests on a technique called megakernel—an inference kernel that fuses hundreds of small programs traditionally scheduled separately into a single large program. By eliminating context-switching overhead and leveraging TPU's unique on-chip memory architecture, megakernel pushes memory bandwidth utilization toward the hardware's theoretical maximum. Combined with DeepSeek's DSpark speculative decoding framework, the result is 709 tokens per second on Kimi K3, and the implementation is now open source.

**Equal hardware, different results**

Inferact benchmarked 16 TPU v7 Ironwood against 16 GB200s under identical conditions—both running Kimi K3, both using the vLLM inference engine, with only the chip and kernel implementation varying. TPU won decisively: 709 tokens per second against 452, a 57% advantage.

DSpark, developed by DeepSeek, accelerates inference by having a small model rapidly predict candidate tokens, then batching large-model verification. Inferact implemented this pipeline on TPU, achieving an acceptance length of 6 (averaging six verified tokens per guess) with single-step decode latency around 8.5 milliseconds.

Without speculative decoding, the gap widens further. At batch size 1, TPU reached 249 tokens per second versus 127 on GB200—nearly a 2× difference. At batch size 8, TPU hit 865 tokens per second while GB200 managed 636. On Qwen 27B with four TPUs, Inferact achieved 1,515 tokens per second against 695 for the equivalent GB200 configuration.

The speed advantage came without accuracy loss. Using greedy decoding validation, Kimi K3 on TPU scored 94.4% on GPQA-Diamond and 97.2% on GSM8K—identical to GPU results.

**The paradox of hardware specs**

The 57% gap cannot be explained by the chips' raw specifications. TPU v7's HBM bandwidth measures 7,380 GB/s; GB200 achieves even higher at 8,000 GB/s. Yet the chip with lower bandwidth delivered higher throughput—a reversal pointing to software rather than silicon.

Traditional inference workflows split computation across hundreds of independent kernels, each handling a narrow operation. Between kernel transitions, gaps accumulate where memory bandwidth sits idle. CUDA Graphs and Nvidia's PDL technology narrow these gaps, but kernel boundaries persist.

Megakernel eliminates the boundary entirely. For Kimi K3's 92 MoE layers, Inferact compiled all 92 into a single Pallas program—a single function call executes the complete forward pass. With boundaries gone, cross-layer weight prefetching becomes seamless: while layer N computes its MoE operations, layer N+1's attention weights already prefetch from main memory into on-chip cache. Computation and data movement overlap; memory bandwidth rarely idles.

TPU's architecture enabled this approach. Each TPU v7 TensorCore carries 64 MiB of VMEM on-chip memory under complete software control—engineers decide precisely when to load which data and when to free the space. This capacity accommodates both the current layer's computation data and the next layer's prefetched weights. GPU architecture (Blackwell) distributes roughly 38 MiB of on-chip memory across 152 SMs with automatic hardware scheduling, leaving less room for such fine-grained orchestration.

Inferact hand-coded the megakernel in Pallas, Google's low-level TPU programming language (comparable to CUDA for Nvidia GPUs). TPU's default XLA compiler excels at single-layer optimization, but 92-layer coordination presents too many decision branches for automatic scheduling to find the optimal path. By hand-writing the full pipeline in Pallas, engineers controlled each data movement decision. A secondary benefit: compilation time dropped from 30+ minutes to under 90 seconds, enabling rapid iteration—kernel changes verified in approximately 90 seconds.

Currently, megakernel is custom-optimized for Kimi K3's architecture; other model layouts require separate adaptation. Inferact has indicated plans to extend megakernel support across additional model architectures.

**vLLM's creators turn to TPU**

Inferact was founded in November 2025 by the original vLLM developers—the team that built Nvidia GPU inference to prominence. They launched officially in January and immediately demonstrated that TPU inference now outpaces GPU.

vLLM is the heavyweight of open-source inference engines: it supports over 500 model architectures, boasts more than 2,000 community contributors, and powers production inference at Meta, Google, and Character.ai.

CEO Simon Mo, the original vLLM project maintainer and Berkeley EECS graduate, previously worked at Anyscale. Co-founder Woosuk Kwon, who initiated the entire vLLM project and holds a Berkeley PhD in Computer Science under Ion Stoica's supervision, authored PagedAttention—an algorithm that solved GPU memory fragmentation and remains vLLM's core innovation. Chief Scientist Kai You, recipient of Tsinghua University's special scholarship, spearheaded vLLM's distributed inference capabilities.

This TPU megakernel effort emerged from a joint engineering collaboration between Inferact and Google Cloud, aimed at making TPU a first-class vLLM target. All optimizations will be contributed upstream to the open-source community; the tpu-megakernels repository represents the first public outcome of this partnership.