Analysis of architectural and cost tradeoffs between general-purpose GPUs and custom ASICs for AI inference workloads.
Analysis of the GPU and ASIC landscape shows significant progress in custom silicon performance relative to merchant solutions. Google and Broadcom's TPU is rapidly closing the competitive gap with Nvidia's offerings on compute cost performance. The analysis demonstrates a ~70% reduction in cost per token from TPU v6 to TPU v7, bringing performance to parity or below Nvidia's GB200 NVL72 in terms of absolute cost. Despite this progress, Nvidia maintains competitive advantages through faster time to market and its CUDA software ecosystem, which remains a key moat for enterprise customers. In contrast, Amazon's Trainium and AMD solutions have delivered only ~30% cost reduction and currently lag behind Nvidia and Google solutions on absolute cost.
TPU adoption is accelerating among customers with strong software capabilities and for internal workloads. Google's increased deployment of TPUs for training its Gemini models reflects confidence in the platform's performance trajectory. Anthropic has emerged as a major customer, placing orders worth $21 billion with Broadcom with shipments expected in mid-2026. This substantial commitment underscores the viability of ASIC solutions for organizations with the internal software talent to develop on TPU platforms.
AMD and Amazon currently face steeper competitive challenges in the inference cost curve. While AMD's MI455X rack and Trainium 3 and 4 are expected to be more competitive with Nvidia and Google solutions in late 2026, their current offerings lag on both generation-to-generation cost reductions and absolute inference cost. However, developments since AWS re:Invent suggest that Trainium 3 and 4 may offer stronger performance that rectifies some of Trainium 2's challenges, with AMD claiming its rack-level solution will match Nvidia's VR200 for training and inference applications.
Compute dies are already pushing reticle limits, shifting the focus to adjacent technologies as the primary drivers of future cost reduction. Advancements in networking, memory, and packaging technologies are expected to enable the next phase of cost improvements as raw compute performance reaches physical boundaries. Nvidia is positioned strongly in these areas, significantly outspending competitors on R&D, maintaining leadership in networking through its Mellanox business, and advancing memory technology with its context memory storage controller offering. Broadcom similarly leads with best-in-class processor and accelerator solutions paired with industry-leading ethernet networking and SERDES capabilities.
The accelerator market is expected to evolve toward selective hyperscaler deployment of custom ASICs for external customers alongside internal workloads. While Nvidia is expected to maintain near-term market leadership through its innovation pace and software ecosystem, Google and Broadcom's TPUs are positioned to capture share from customers with strong software capabilities and from Google's own substantial internal demand. The competitive dynamics will increasingly depend on execution in networking, memory, and packaging innovation, areas where Nvidia and Broadcom demonstrate the strongest current positioning.