NVIDIA demonstrates its RC18 inference processor, a specialized chip for AI model inference workloads.

While it would be convenient for both chip makers and those running machine learning workloads if training and inference could be done on the same device, Nvidia currently maintains a virtual lock on machine learning training with its Tesla "Volta" GPU accelerators. The company has also made significant inroads in the nascent inference space with its "Turing" GPUs, which are used in its most recent GeForce graphics cards as well as a low-voltage Tesla variant aimed at inference offload in servers. At the recent Hot Chips 31 conference at Stanford University, we spoke with Bill Dally, who has headed up Nvidia Research for the past decade following a long career at CalTech, MIT, and Stanford designing supercomputers and interconnects. Dally has co-authored three books on interconnects: "Digital Systems Engineering" (1998), "Principles and Practices of Interconnection Networks" (2004), and "Digital Design: A Systems Approach" (2012).

Dally's team unveiled the RC18 research chip at the Hot Chips conference, designed to demonstrate efficient inference acceleration at 128 teraops. According to Dally, "The RC18 chip is an efficient inference engine – it's 9.5 teraops per watt, done in vanilla 16 nanometer process from TSMC." This equates to approximately 13.5 watts, a power profile and performance efficiency that appeal to hyperscalers and cloud builders seeking inference engines. Dally emphasized that the chip can replace the Nvidia Deep Learning Accelerator, noting it is "actually designed very similarly to NVDLA, but we are as interested in the tools as much as we are about the chip." The team developed design space exploration tools that test "all combinations of vector widths, the number of vector units, the sizes of buffer arrays, and ways of tiling the loop to stage things," with the tools identifying optimal configurations for various neural networks.

A critical question concerns optimal matrix sizes for vector operations. Google's TPU1 used 256×256 matrices while its TPU3 reduced this to 128×128, and Intel's Nervana inference chips operate at 32×32. The RC18 employs 8×8 matrices. Dally explained the fundamental tradeoff: "As you make it bigger it gets slightly more efficient on really big matrix multiplies but it gets less efficient when it doesn't fit. If you make it really big, you amortize out the external overhead more but then if you can't fill it you are just wasting space." For inference, Dally sees the optimal range as "somewhere between 8×8 and 16×16 – and it's nice to make it powers of two because it makes everything map." He predicted that newer networks may demand smaller granularity, stating "I think it's going to stay in the 8×8 to 16×16 range."

The RC18 incorporates innovations aimed at power efficiency. Rather than running all processing elements from a single clock—which requires slowing to accommodate the chip's lowest-voltage regions—each processing element runs on "a separate clock which is locally generated and tracking the power supply." This allows the chip to "run a frequency that's right up to the hairy edge and then throttle back as the power supply dips." However, this requires fast synchronization between clock domains. Dally's team created synchronizers achieving "on average in half a clock cycle" synchronization, compared to the typical "five or six clock cycles to go from one clock domain to another." The technology uses asynchronous communication that "only when you're in that hairy region does it then delay you an additional clock to make it safe. But that almost never happens, so on average it ends up being a half clock cycle." This synchronization technology won the best paper award at the International Symposium on Asynchronous Circuits and Systems (ASYNC) in May.