Machine Learning Trends
Our Trends dashboard offers curated key numbers, visualizations, and insights that showcase the significant growth and impact of artificial intelligence.
Last updated on Apr 09, 2024
Display growth values in:
Compute Trends
We’ve compiled a dataset of the training compute for over 120 machine learning models, highlighting novel trends and insights into the development of AI since 1952, and what to expect going forward.
Data Trends
Based on our previous analysis of trends in dataset size, we project the growth of dataset size in the language and vision domains. We explore the limits of this trend by estimating the total stock of available unlabeled data over the next decades.
Hardware Trends
We analyze recent trends in machine learning hardware performance, focusing on metrics such as computational performance, memory, interconnect bandwidth, price-performance, and energy efficiency across different GPUs and accelerators. The analysis aims to provide a holistic view of ML hardware capability and bottlenecks.
Algorithmic Progress
We study how algorithmic improvements and increases in computational power have improved the performance of language models from 2014 to 2024. We find that the progress from new algorithms surpasses what we’d expect from merely increasing our computing resources, occurring at a pace equivalent to doubling computational power every 5 to 14 months.
Investment Trends
We combine training compute and GPU price-performance data to estimate the cost of compute in US dollars for the final training run of 124 machine learning systems published between 2009 and 2022, and find that the cost has grown by approximately 0.5 orders of magnitude per year.
Biological Models
The White House recently issued an Executive Order requiring enhanced oversight for AI models trained on biological data exceeding 1e23 operations. We provide an overview of our expanded data coverage to biological sequence models, revealing a significant increase in computational resources and the extensive availability of protein and DNA sequence data. Our analysis identifies critical trends in training compute, data stock, and potential regulatory gaps.
Acknowledgements
We thank Tom Davidson, Lukas Finnveden, Charlie Giattino, Zach-Stein Perlman, Misha Yagudin, Robi Rahman, Jai Vipra, Patrick Levermore, Carl Shulman, Ben Bucknall and Daniel Kokotajlo for their feedback.
Several people have contributed to the design and maintenance of this dashboard, including Jaime Sevilla, Pablo Villalobos, Anson Ho, Tamay Besiroglu, Ege Erdil, Ben Cottier, Matthew Barnett, David Owen, Robi Rahman, Lennart Heim, Marius Hobbhahn, David Atkinson, Keith Wynroe, Christopher Phenicie, Nicole Maug, Alex Haase, Robert Sandler and Edu Roldan.
Citation
Cite this work as
Epoch AI (2023), "Key Trends and Figures in Machine Learning". Published online at epochai.org. Retrieved from: 'https://epochai.org/trends' [online resource]
BibTeX citation
@misc{epoch2023aitrends,
title = "Key Trends and Figures in Machine Learning",
author = {Epoch AI},
year = 2023,
url = {https://epochai.org/trends},
note = "Accessed: "
}