Forbes analyzes whether Google TPU performance in AI inference validates custom silicon as NVIDIA GPU alternative

After nearly a year since the introduction of the Google TensorFlow Processing Unit, or TPU, Google released detailed performance and power metrics for its in-house AI chip. The chip was developed to address a fundamental challenge: Google foresaw a dramatic shift in its computing needs towards supporting Machine Learning workloads, which are profoundly compute intensive. Continuing to use Intel CPUs was cost prohibitive and would not meet Google's needs for rapid response times across millions of simultaneous users and queries. Google had been using NVIDIA GPUs for training the underlying neural networks that allow machines to recognize patterns in data, and x86 CPUs to execute the queries across these neural networks, called inferencing. The company decided to develop a chip that could handle inference at a lower cost, with higher performance, while consuming far less power.

Google released extensive architectural details and performance data comparing the TPU with the generation of NVIDIA and Intel chips it had at its facility at the time—Intel's Haswell, which is 3 generations old, and the NVIDIA Kepler, architected in 2009, long before anyone was using GPUs for machine learning. NVIDIA CEO Jensen Huang provided updated comparisons based on NVIDIA's latest generation PASCAL chips, which produced very different results. While NVIDIA's initial deficit appeared to be only 1/13th the performance of the TPU, comparing current generation chips showed NVIDIA with a 2X advantage, albeit at 3x the power consumption. The TPU screams at 90 trillion operations per second, nearly twice that of the GPU, and consumes only 1/3rd the power. The P40 has strong floating point capabilities useful in training and greater memory bandwidth, while the TPU is designed to do one thing extremely well: multiply tensors (integer matrices) in parallel that are used to represent the deep neural networks used in Machine Learning for AI.

What matters most is the proof that any serious work being done on AI requires serious acceleration—whether by a GPU, an FPGA, an ASIC or a many-core CPU—all of which will need to be at least 2 orders of magnitude faster than a traditional Intel Xeon server CPU if AI is to be affordable and responsive. Google's investment in custom silicon demonstrates that processing Machine Learning is a sufficiently large and vital workload to justify optimizing its datacenters for ML.

Users of Google Machine Learning services directly benefit as more services move to run on TPU. Google has lowered the price of selected services by as much as 6x, directly attributing the savings to the TPU. Google wins by having a more competitive platform for internal use and cloud ML services while saving on CAPEX and power consumption for its massive datacenters.

The TPU does not represent a direct or immediate risk to silicon vendors such as Intel and NVIDIA. Most inference work today is done by Intel Xeon CPUs in the datacenter and ARM CPUs at the edge, deployed at a more modest scale than seen at Google. Google is still using NVIDIA GPUs for training its neural networks, so the TPU has not taken a significant chunk out of NVIDIA's business. Additionally, the TPU is only available to Google's internal data scientists and users of Google's AI cloud services. Google Cloud remains a distant third to Amazon Web Services and Microsoft Azure, both of whom offer NVIDIA GPUs in their cloud services for Machine Learning applications. The world's largest datacenters—Amazon, Alibaba, Baidu, Facebook, Microsoft and TenCent—will continue to purchase GPUs and FPGAs for their acceleration workloads.