Epoch is a research group trying to forecast the development of transformative Artificial Intelligence. We try to understand how progress in AI happens and what economic impacts we might see from advanced AI.
We want to enable better governance during this economic transition by informing about the timing of new developments and what levers can be used to influence AI progress.
Our research agenda includes both empirical and theoretical work.
Modelling the future of AI
We are developing statistical and economic models to answer crucial questions about the direction and impact of AI in the coming decades.
Studying trends in Machine Learning
Curating and analyzing information about significant ML systems like compute budgets, training dataset sizes, cost and performance.
Algorithmic progress in computer vision
Dec. 12, 2022
Ege Erdil, Tamay Besiroglu
We use a dataset of over a hundred computer vision models from the last decade to investigate how better algorithms and architectures have enabled researchers to use compute and data...
Predicting GPU performance
Dec. 01, 2022
Marius Hobbhahn and Tamay Besiroglu
We develop a simple model that predicts progress in the performance of field-effect transistor-based GPUs under the assumption that transistors can no longer miniaturize after scaling down to roughly the...
Will we run out of ML data? Evidence from projecting dataset size trends
Oct. 10, 2022
Pablo Villalobos, Jaime Sevilla, Lennart Heim, Tamay Besiroglu, Marius Hobbhahn and Anson Ho
Based on our previous analysis of trends in dataset size, we project the growth of dataset size in the language and vision domains. We explore the limits of this trend...
The longest training run
Aug. 17, 2022
Jaime Sevilla, Tamay Besiroglu, Owen Dudney and Anson Ho
Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore...
Machine Learning Model Sizes and the Parameter Gap
Jul. 05, 2022
Pablo Villalobos, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, Anson Ho and Marius Hobbhahn
The model size of notable Machine Learning systems has grown ten times faster than before since 2018. After 2020 growth has not been entirely continuous: there was a jump of...
Projecting compute trends in Machine Learning
Mar. 07, 2022
Tamay Besiroglu, Lennart Heim and Jaime Sevilla
Projecting forward 70 years worth of trends in the amount of compute used to train Machine Learning models.
Compute Trends Across Three Eras of Machine Learning
Feb. 11, 2022
Jaime Sevilla, Tamay Besiroglu, Anson Ho, Lennart Heim, Marius Hobbhahn, and Pablo Villalobos
Compute, data, and algorithmic advances are the three fundamental factors that guide the progress of modern Machine Learning (ML). In this paper we study trends in the most readily quantified...
Estimating Training Compute of Deep Learning Models
Jan. 20, 2022
Jaime Sevilla, Lennart Heim, Marius Hobbhahn, Tamay Besiroglu, Anson Ho and Pablo Villalobos
We describe two approaches for estimating the training compute of Deep Learning systems, by counting operations and looking at GPU time.
Parameters, Compute and Data Trends in Machine Learning
Jaime Sevilla et al.
Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks
Tilman Rauker, Anson Ho, Stephen Casper, and Dylan Hadfield-Menell
The last decade of machine learning has seen drastic increases in scale and capabilities, and deep neural networks (DNNs) are increasingly being deployed across a wide range of domains. However,...