ML trends

Trends

Our Trends section features key numbers and data visualizations in AI, links to our reports and other sources that showcase the change and growth in AI over time. Last updated on Jul 17, 2023.

Display growth values in:

Training compute growth rate

Likely

0.62 OOM/year

Growth rate in the training compute of milestone systems since 2010.

Training
data

Plausible

2024

Median projected year we run out of high quality text.

Most parameters in a dense model

Uncertain

540 billion

In Google’s PaLM and Minerva.

GPU price-performance

Likely

0.122 OOM/year

Growth rate in the amount of buyable FLOP/s per dollar for FP32 precision.

Algorithmic improvements

Plausible

0.4 OOM/year

Decline rate in the physical compute requirements for image classification.

Training costs growth rate

Likely

0.49 OOM/year

Growth rate in USD of the cost of the most expensive training run since 2009.

Compute Trends

Deep Learning compute growth

Likely

0.62 OOM/year

Growth rate in the training compute of milestone systems since 2010.

Pre-Deep Learning compute growth

Likely

0.18 OOM/year

Growth rate in the training compute of milestone systems from 1956 to 2010.

Large-scale vs regular scale

Uncertain

100x

Gap between the compute used by large-scale models and other notable models in the same year.

Paper

Training compute of milestone Machine Learning systems over time

We compile the largest known dataset of milestone Machine Learning models to date.
Training compute grew by 0.2 OOM/year up until the Deep Learning revolution around 2010, after which growth rates increased to 0.6 OOM/year.
We also find a new trend of “large-scale” models that emerged in 2016, trained with 2-3 OOMs more compute than other systems in the same period.

Data Trends

When might we run out of high-quality text?

Plausible

2024

Median projected year in which the training dataset of notable ML models exhausts the stock of professionally edited texts on the internet.

When might we run out of text altogether?

Uncertain

2040

Median projected year in which the training dataset of notable ML models exhausts the stock of all text on the internet.

Paper

Will we run out of ML data? Evidence from projecting dataset size trends

The available stock of text and image data grew by 0.14 OOM/year between 1990 and 2018, but has since slowed to 0.03 OOM/year.
At current rates of data production, our projections suggest that we will run out of high-quality text, low-quality text, and images by 2024, 2040 and 2046 respectively.

Largest training dataset

Uncertain

1.87 trillion words

Training dataset size of FLAN, the most data-intensive language model to date.

Stock of data on the internet

Plausible

100 trillion words

Number of words in Common Crawl, the largest public repository of web crawl data on the internet.

Model Size Trends

Parameter count growth rate

Plausible

0.45 OOM/year

Growth rate in the number of trainable parameters of notable ML models since 2018.

The parameter gap

Uncertain

20B to 70B

Range of parameter counts in which there was a statistically significant absence of milestone models.

Paper

Machine Learning model sizes and the parameter gap

Between the 1950s and 2018, model sizes grew at a rate of 0.1 OOM/year, but this rate accelerated dramatically after 2018.
This is partly due to a statistically significant absence of milestone models with between 20 billion and 70 billion parameters, which we call the “parameter gap.”

Largest model trained end-to-end

Uncertain

540 billion parameters

Parameter count of PaLM and Minerva, the largest dense models trained end-to-end to date.

Hardware Trends

GPU price-performance growth rate

Likely

0.122 OOM/year

Growth rate in the amount of buyable FLOP/s per dollar for FP32 precision.

Report

Trends in GPU price-performance

We consider trends in GPU price-performance between 2006 and 2022.
We find that the FLOP per second per $ for GPUs at FP32 precision have increased at a rate of 0.12 OOM/year on average.
This trend is instead 0.15 OOM/year for Machine Learning-specific GPUs, and 0.10 OOM/year for state of the art GPUs.

Highest GPU price-performance

Uncertain

4.2e18 FLOP per $

Highest price-performance of a consumer GPU, from the AMD Radeon RX 7900 XTX.

Algorithmic Progress

Compute-efficiency growth in computer vision

Plausible

0.4 OOM/year

Decline rate in the physical compute required to achieve a given performance on ImageNet since 2012.

Data-efficiency growth in computer vision

Very uncertain

0.1 OOM/year

Decline rate in the dataset size required to achieve a given performance on ImageNet since 2012.

Paper

Revisiting algorithmic progress

Algorithmic progress explains roughly 45% of performance improvements in image classification, and most of this occurs through improving compute-efficiency.
The amount of compute needed to achieve state-of-the-art performance in image classification on ImageNet declined at a rate of 0.4 OOM/year in the period between 2012 and 2022, faster than prior estimates suggested.

Chinchilla scaling laws

Plausible

20 tokens per parameter

Ratio of data to parameters to achieve compute-optimal scaling for LLMs, according to (Hoffmann et al, 2022)

Investment Trends

Training costs growth rate

Likely

0.49 OOM/year

Growth rate in USD of the cost of the most expensive training run since 2009.

Training costs for large-scale models growth rate

Plausible

0.2 OOM/year

Growth rate in USD of the cost of training “large-scale” systems since 2015.

Report

Trends in the dollar training cost of Machine Learning systems

The dollar cost for the final training run of milestone ML systems increased at a rate of 0.5 OOM/year between 2009 and 2022.
Since September 2015, the cost for “large-scale” systems (systems that used a relatively large amount of compute) has grown more slowly, at a rate of 0.2 OOM/year.

Most expensive training run

Plausible

$50 million

Training cost of the compute for the final training run of GPT-4, the ML system with the most expensive training requirements to date.

Acknowledgements

We thank Tom Davidson, Lukas Finnveden, Charlie Giattino, Zach-Stein Perlman, Misha Yagudin, Robi Rahman, Jai Vipra, Patrick Levermore, Carl Shulman, Ben Bucknall and Daniel Kokotajlo for their feedback.

Citation

Cite this work as

  Epoch (2023), "Key trends and figures in Machine Learning". Published online at epochai.org. Retrieved from: 'https://epochai.org/trends' [online resource]

BibTeX citation

@misc{epoch2023mltrends,
  title = "Key trends and figures in Machine Learning",
  author = {Epoch},
  year = 2023,
  url = {https://epochai.org/trends},
  note = "Accessed: "
}

If you spot an error or would like to provide feedback, please reach out at .