Trends in Machine Learning Hardware

## Executive summary

Peak computational performance of ML hardware for different precisions

OP/s

FP64

FP32

FP16

tensor-FP32/TF32

tensor-FP16

tensor-INT8

tensor-INT4

NVIDIA A100

Google TPU v4

NVIDIA H100 SXM

1011​1012​1013​1014​1015​FP32FP16FP32 and FP64 supportedEmergence of new number formatsYear20102012201420162018202020222024

![Epoch AI](https://epoch.ai/assets/logo/epoch-full-standard.svg)\|CC-BYepoch.aiDownload graph

**Figure 1:** Peak computational performance of common ML accelerators at a given precision. New number formats have emerged since 2016. Trendlines are shown for number formats with eight or more accelerators: FP32, FP16 (FP = floating-point, tensor-\* = processed by a tensor core, TF = Nvidia tensor floating-point, INT = integer)

We study the performance of GPUs for computational performance across different number representations, memory capacities and bandwidth, and interconnect bandwidth using a dataset of 47 ML accelerators (GPUs and other AI chips) commonly used in ML experiments from 2010-2023, plus 1,948 additional GPUs from 2006-2021. Our main findings are:

1. Lower-precision number formats like 16-bit floating point (FP16) and 8-bit integers (INT8), combined with specialized tensor core units, can provide order-of-magnitude performance improvements for machine learning workloads compared to traditionally used 32-bit floating point (FP32). For example, we estimate, though using limited amounts of data, that using tensor-FP16 can provide roughly 10x speedup compared to FP32.
2. Given that the overall performance of large hardware clusters for state-of-the-art ML model training and inference depends on factors beyond just computational performance, we investigate memory capacity, memory bandwidth and interconnects, and find that:
1. Memory capacity is doubling every ~4 years and memory bandwidth every ~4.1 years. They have increased at a slower rate than computational performance which doubles every ~2.3 years. This is a common finding and often described as the _memory wall_.
2. The latest ML hardware often comes with proprietary chip-to-chip interconnect protocols (Nvidia’s NVLink or Google’s TPU’s ICI) that offer higher communication bandwidth between chips compared to the PCI Express (PCIe). For example, NVLink in H100 supports 7x the bandwidth of PCIe 5.0.
3. Key hardware performance metrics and their improvement rates found in the analysis include: computational performance \[FLOP/s\] doubling every 2.3 years for both ML and general GPUs; computational price-performance \[FLOP per $\] doubling every 2.1 years for ML GPUs and 2.5 years for general GPUs; and energy efficiency \[FLOP/s per Watt\] doubling every 3.0 years for ML GPUs and 2.7 years for general GPUs.

|  | Specification and unit | Growth rate<br>Doubling time10x timeOOMs per year | Datapoint of highest performance<br>Metric prefixScientific notation | N |
| --- | --- | --- | --- | --- |
| Computational Performance | FLOP/s (FP32) | 2x eve

... (truncated, 60 KB total)