Besiroglu et al.
paperAuthors
Credibility Rating
Good quality. Reputable source with community review or editorial standards, but less rigorous than peer-reviewed venues.
Rating inherited from publication venue: arXiv
Empirical analysis of progress in computer vision that decomposes improvements into contributions from compute scaling, data scaling, and algorithmic advances using Shapley values, relevant for understanding AI capability development trajectories.
Paper Details
Metadata
Abstract
We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural scaling laws, and infer a decomposition of progress into the scaling of compute, data, and algorithms. Using Shapley values to attribute performance improvements, we find that algorithmic improvements have been roughly as important as the scaling of compute for progress computer vision. Our estimates indicate that algorithmic innovations mostly take the form of compute-augmenting algorithmic advances (which enable researchers to get better performance from less compute), not data-augmenting algorithmic advances. We find that compute-augmenting algorithmic advances are made at a pace more than twice as fast as the rate usually associated with Moore's law. In particular, we estimate that compute-augmenting innovations halve compute requirements every nine months (95\% confidence interval: 4 to 25 months).
Summary
This paper analyzes algorithmic progress in image classification on ImageNet by decomposing performance improvements into contributions from compute scaling, data scaling, and algorithmic innovations. Using Shapley values and neural scaling law models, the authors find that algorithmic improvements have been roughly as important as compute scaling for progress in computer vision. Notably, most algorithmic advances are compute-augmenting (enabling better performance with less compute) rather than data-augmenting, and these compute-augmenting innovations occur at a rate exceeding Moore's law, with compute requirements halving approximately every nine months.
Cited by 1 page
| Page | Type | Quality |
|---|---|---|
| Epoch AI | Organization | 51.0 |
Cached Content Preview
# Algorithmic progress in computer vision
Ege Erdil∗Tamay Besiroglu∗†
(∗Epoch
†MIT FutureTech
)
###### Abstract
We investigate algorithmic progress in image classification on ImageNet, perhaps the most well-known test bed for computer vision. We estimate a model, informed by work on neural scaling laws, and infer a decomposition of progress into the scaling of compute, data, and algorithms. Using Shapley values to attribute performance improvements, we find that algorithmic improvements have been roughly as important as the scaling of compute for progress computer vision. Our estimates indicate that algorithmic innovations mostly take the form of compute-augmenting algorithmic advances (which enable researchers to get better performance from less compute), not data-augmenting algorithmic advances. We find that compute-augmenting algorithmic advances are made at a pace more than twice as fast as the rate usually associated with Moore’s law. In particular, we estimate that compute-augmenting innovations halve compute requirements every nine months (95% confidence interval: 4 to 25 months).
††You can reproduce all our results using the code presented in [in this Colab notebook](https://colab.research.google.com/drive/1-gBOhcVaYDNgO-9_EAOanfH7AeD17EGv?usp=sharing "").††We thank Matthew Barnett, Neil Thompson, Pablo Villalobos, Jaime Sevilla, Danny Hernandez, Marius Hobbhahn, Adam Papineau, Rick Korzekwa, and Lawrence Phillips for their helpful comments. We are also grateful to Owen Dudney for his assistance in building the relevant data-sets.
## 1 Introduction
It is a matter of debate how much of the recent progress in machine learning has come from the scaling of compute and the sizes of models and data-sets, and how much has come from improvements in the underlying algorithms and architectures. In this work, we provide a decomposition of these different sources of progress on the task of image classification on the well-known ImageNet dataset, and provide insights into how algorithmic advances produce better models.
(a) Pareto frontiers in data and compute for AlexNet performance
(b) Pareto frontiers in data and compute for ResNeXt-101 performance
(c) Pareto frontiers in data and compute for ViT-e performance
Figure 1: \*
Figure 1. Pareto frontiers for training models to achieve performance of well-known models over time. Our estimates indicate that compute-augmenting algorithmic improvements double the effective compute available to train image classification models every nine months. Note that these plots are predictions made by a model, and could be extrapolating the model beyond its domain of validity (see [3.1](https://ar5iv.labs.arxiv.org/html/2212.05153#S3.SS1 "3.1 Empirical model ‣ 3 Empirical a
... (truncated, 57 KB total)b5fab0db54c83703 | Stable ID: ODQ4N2FhNT