Google's TPU v2 (Tensor Processing Unit second generation) is designed to accelerate both training and inference AI workloads with significantly improved performance over v1.

Google shared details about its original TPU (Tensor Processor Unit) and the Cloud TPU, otherwise known as TPU v2, at Hot Chips 2017. The company designed its own ASIC in a skunkworks-class project after becoming frustrated with the slowing pace of processor development. Google believed Moore's Law was either in a slowdown or near retirement, and since Dennard scaling already expired around 2006 due to increased current leakage, the company no longer saw massive clock frequency increases with newer processors. With its user base growing daily and large price reductions and massive performance gains unavailable off the shelf, Google stepped into the processor design realm to avoid growing its data center footprint.

The first generation TPU, designed to perform inference only, featured a single core with a 256x256x8b Mixed Multiply Unit (MXU) that provided up to 45 TFLOPS. The design process began in 2013 with the theme of "go fast and run with scissors," and fourteen months later, the TPU made its debut in Google's data centers after remaining deployed for a year before disclosure. The TPU was built on a 28nm ASIC that consumed 40W under load and ran at 700MHz. The engineers maintained backward compatibility with existing infrastructure by designing the TPU to fit into a standard 3.5" server HDD bay and connect via a custom PCIe 3.0 x16 connector to a backplane. The ASIC minimized code by running without its own branch processor, relying instead on the host to feed it instructions. The TPU supported only 11 instructions, with just five used to execute workloads, and its 256x256 Matrix Multiply Unit processed hundreds of thousands of operations per clock cycle.

Google's Cloud TPU represented a significant evolution, supporting both training and inference workloads with dual cores featuring one 128x128x32b MXU apiece. Each cloud TPU board consisted of four ASICs on a single motherboard with 64GB of HBM providing 2,400 GBps of memory bandwidth, generating up to 180 TFLOPS per board. Google deployed 64 Cloud TPUs into a single "pod" that generated up to 11.5 petaFLOPS of compute and wielded 4 TB of HBM. The ASICs rode on a motherboard with large heatsinks and eight networking connectors on the edge of the board, widely thought to be Intel's Omni-Path and IBM's BlueLink.

Google began offering Cloud TPU services to the public through its Google Cloud Platform, with each Virtual Machine instance coming with one TPU attached. The company also donated a thousand of the units to top researchers. Cloud TPUs were programmable with TensorFlow, which helped speed adoption. Google claimed the processors offered 50% more performance than the first-generation model, though the company had not yet provided efficiency measurements.