Compute Trends
Training compute of milestone Machine Learning systems over time
- We compile the largest known dataset of milestone Machine Learning models to date.
- Training compute grew by 0.2 OOM/year up until the Deep Learning revolution around 2010, after which growth rates increased to 0.6 OOM/year.
- We also find a new trend of “large-scale” models that emerged in 2016, trained with 2-3 OOMs more compute than other systems in the same period.
Data Trends
Will we run out of ML data? Evidence from projecting dataset size trends
- The available stock of text and image data grew by 0.14 OOM/year between 1990 and 2018, but has since slowed to 0.03 OOM/year.
- At current rates of data production, our projections suggest that we will run out of high-quality text, low-quality text, and images by 2024, 2040 and 2046 respectively.
Model Size Trends
Machine Learning model sizes and the parameter gap
- Between the 1950s and 2018, model sizes grew at a rate of 0.1 OOM/year, but this rate accelerated dramatically after 2018.
- This is partly due to a statistically significant absence of milestone models with between 20 billion and 70 billion parameters, which we call the “parameter gap.”
Hardware Trends
Trends in GPU price-performance
- We consider trends in GPU price-performance between 2006 and 2022.
- We find that the FLOP per second per $ for GPUs at FP32 precision have increased at a rate of 0.12 OOM/year on average.
- This trend is instead 0.15 OOM/year for Machine Learning-specific GPUs, and 0.10 OOM/year for state of the art GPUs.
Algorithmic Progress
Revisiting algorithmic progress
-
Algorithmic progress explains roughly 45% of performance improvements in image classification, and most of this occurs through improving compute-efficiency.
-
The amount of compute needed to achieve state-of-the-art performance in image classification on ImageNet declined at a rate of 0.4 OOM/year in the period between 2012 and 2022, faster than prior estimates suggested.
Investment Trends
Trends in the dollar training cost of Machine Learning systems
- The dollar cost for the final training run of milestone ML systems increased at a rate of 0.5 OOM/year between 2009 and 2022.
- Since September 2015, the cost for “large-scale” systems (systems that used a relatively large amount of compute) has grown more slowly, at a rate of 0.2 OOM/year.
Acknowledgements
We thank Tom Davidson, Lukas Finnveden, Charlie Giattino, Zach-Stein Perlman, Misha Yagudin, Robi Rahman, Jai Vipra, Patrick Levermore, Carl Shulman, Ben Bucknall and Daniel Kokotajlo for their feedback.
Citation
Cite this work as
Epoch (2023), "Key trends and figures in Machine Learning". Published online at epochai.org. Retrieved from: 'https://epochai.org/trends' [online resource]
BibTeX citation
@misc{epoch2023mltrends,
title = "Key trends and figures in Machine Learning",
author = {Epoch},
year = 2023,
url = {https://epochai.org/trends},
note = "Accessed: "
}