Notable AI Models Documentation

Epoch's AI Notable AI Models Database is a collection of historically significant or cutting-edge machine learning models for research about trends in the history and future of artificial intelligence.

Overview

Epoch AI’s Notable AI Models Database is a collection of historically significant or cutting-edge machine learning models, and key information about their training. This database is useful for research about trends in the history and future of artificial intelligence.

This documentation describes which models are contained within the database, its records (including data fields and definitions), processes for adding new entries and auditing accuracy. It also includes a changelog and acknowledgements.

The database is available on our website as a visualization or table, and is available for download as a daily-updated CSV file. For a quick-start example of loading the data and working with it in your research, see this Google Colab demo notebook.

If you would like to ask any questions about the database, or suggest a model that should be added, feel free to contact us at data@epochai.org.

If this database is useful for you, please cite it.

Citation

Epoch AI, ‘Parameter, Compute and Data Trends in Machine Learning’. Published online at epochai.org. Retrieved from: ‘https://epochai.org/data/notable-ai-models’ [online resource]

BibTeX citation

@misc{epoch2022pcdtrends,
  title = "Parameter, Compute and Data Trends in Machine Learning",
  author = {{Epoch AI}},
  year = 2022,
  url = {https://epochai.org/data/notable-ai-models},
  note = "Accessed: "
}

Inclusion

The database focuses on notable ML models: models that have advanced the state of the art, had a large influence in the field’s history, or had a large impact within the world. Here, we detail criteria for inclusion, and give an overview of how the data have been collected.


Criteria

To be included in the database, an ML model must satisfy all inclusion criteria:

  • there must be reliable documentation of its existence and relevance to machine learning;
  • the model must include a learning component, it cannot be a non-learned algorithm;
  • the model must actually have been trained, it cannot be a theoretical description without experimental results;
  • the model must be notable, per any of the notability criteria defined below.

Notability

Models are notable if they satisfy any of the following:

  • highly cited (over 1000 citations);
  • large training cost (over $5,000,000, measured in 2023 USD);
  • significant use (over one million monthly active users);
  • state of the art performance (typically on a recognised ML benchmark, see below for further discussion);
  • indisputable historical significance.

Where there are many related models, for example several checkpoints along training or several sizes of a given model family, the database preferentially includes the version that used the most compute. Other versions may be included where they are notable in their own right.

State of the art

Identifying whether a model is state of the art can be a more involved process, compared to simply checking citations or the training compute budget. We consider a model to be state of the art if there is good reason to believe that it was the best existing model at the time for a task of genuine interest. The default way to provide evidence for this is state of the art performance on a recognised benchmark.

To be recognised, a benchmark should have any of the following:

  • 100+ citations.
  • 10+ submissions in total from 3+ research groups.
  • An associated publication in a reputable peer-reviewed academic venue. The publication does not need to focus exclusively on the benchmark; however, the benchmark should be a key result.

At our discretion, we may also identify models as state of the art where no benchmark result exists, but there is convincing evidence that a model truly is state of the art. Eligible sources of evidence here are comparison on a non-benchmark database, a high-quality user preference study, or demonstration of state of the art capabilities. For example, GraphCast is compared against other weather prediction models on a weather database that is not a standalone benchmark. Nevertheless, we take this as convincing evidence that it is state of the art.

Historical significance

Models can be included on the grounds of historical significance if they marked a significant advance in AI history, even if they did not strictly advance the state of the art on any application. For example, many neural network breakthroughs performed worse than other ML techniques, but were directly influential for later AI development. Evidence to support this status may come from citations in later notable models, discussion in reviews or textbooks, or other unambiguous identification as an influential result.

Table 1: Examples of models evaluated against inclusion criteria.
Example Include? Why
Human-level control through deep reinforcement learning Yes Well-documented learned model, over 1000 citations, advanced state of the art for game play.
Stochastic Neural Analog Reinforcement Calculator Yes No individual associated paper, but other sources confirm its existence, and it was indisputably historically significant as one of the first neural learning systems.
Theory of neural-analog reinforcement systems and its application to the brain model problem No Historically significant, but no experimentally trained models, it’s entirely a theoretical result.
Scaling scaling laws with board games No Doesn’t meet any notability criteria. In addition to not being highly cited and using small compute models, there is no attempt at state of the art results. Rather, this is a paper examining scaling details.

Search process

This database has been collected from a variety of sources: literature reviews, historical accounts of AI development, highly-cited publications from top conferences, high-profile models from leading industry labs, bibliographies of notable papers, pre-existing datasets curating AI papers (see Acknowledgements), and ad hoc suggestions from contributors.

We monitor news coverage, releases from key AI labs, and benchmarks to identify new notable models as they are released. This can lead to a lag for new models. Typically, we aim to add the most prominent releases (e.g. GPT-4) within days of release. For less prominent models, reporting lags may extend to months.


Coverage

As of July 23, 2024, the dataset contains 824 models, 384 of which have compute estimates.

The database does not provide exhaustive coverage of notable models. However, data collection efforts to support Epoch AI research projects have led to more thorough coverage within particular niches, such as models trained with large-scale compute, and biological sequence models.

  • Coverage is most thorough for language and vision models developed since 2018 (254 models and 112 models respectively), albeit with a lag for the newest models. More specialist domains, such as robotics, likely have worse coverage in this period.
  • Coverage is fair, but less thorough, for deep learning language and vision models between 2010-2018 (72 models and 101 models respectively). Again, other domains may have worse coverage.
  • Coverage is quite sparse for historical models before 2010 (143 models before 2010 compared to 681 models after), particularly models outside the paradigm of deep learning. Entries here are focused on notable models mentioned in textbooks and reviews, rather than a systematic search across sources.

If you would like to ask any questions about the database, or suggest a model that should be added, feel free to contact us at data@epochai.org.

Records

The database focuses on information relevant to trends in AI model development. Records in the database have information about three broad areas:

Bibliographic information about the model and its associated publication, for example its title, URL, authors, citations, date, etc.

Training details such as training compute, parameters, dataset size, hardware used for training, etc.

Metadata about the record, such as notes on the above fields with supporting evidence and context, our confidence in the key models, etc.

We provide a comprehensive guide to the database’s fields below. This includes examples taken from Llama-2 70B, one of the best-documented recent models. If you would like to ask any questions about the database, or request a field that should be added, feel free to contact us at data@epochai.org.

Column Type Definition Example from Llama-2 70B Coverage
Abstract Text Abstract text from the publication associated with the model. In this work, we develop and release Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. Our fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. Our models outperform open-source chat models on most benchmarks we tested, and based on our human evaluations for helpfulness and safety, may be a suitable substitute for closedsource models. We provide a detailed description of our approach to fine-tuning and safety improvements of Llama 2-Chat in order to enable the community to build on our work and contribute to the responsible development of LLMs. 47%
384 out of 824 models
Authors Text Comma-separated list of authors.

Authors are named in the way that they report their names in their publications, if applicable. For example, Lê Viết Quốc is credited as “Quoc V. Le” in his publications.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, Thomas Scialom 96%
793 out of 824 models
Base model Categorical (single select) Which base model the model was fine-tuned from, if applicable. [empty]

This is empty because Llama-2 was not finetuned from a base model. For a non-empty example, consider CodeLlama, a Llama-2 finetune. The base model would be Llama-2.
7%
59 out of 824 models
Batch size Numeric Batch size used during training. 4000000 5%
44 out of 824 models
Citations Numeric Number of citations as of last update. Values are collected from Semantic Scholar where available, otherwise manually from Google Scholar. 5474 91%
752 out of 824 models
Confidence Categorical (single select)

Metadata describing our confidence in the recorded values for Training compute, Parameters, and Training dataset size. This describes confidence for the most uncertain of these parameters, where they have a non-empty entry (compute is typically the most uncertain).

The confidence statuses specify 90% confidence that the recorded values are within the following bounds:

  • Confident - ±3x, 0.5 orders of magnitude.
  • Likely - ±10x, 1 order of magnitude.
  • Speculative - ±31x, 1.5 orders of magnitude.

We also provide further statuses:

  • Unknown - we have too little information to even make a speculative estimate.
  • Wrong - we know this estimate is incorrect, and it has been queued for correction.
  • Unverified - this estimate has not yet been assessed for confidence.
Confident 61%
502 out of 824 models
Country (from Organization) Categorical (multiple select)

Country/countries associated with the developing organization(s). Multinational is used to mark organizations associated with multiple countries.

United States of America 99%
819 out of 824 models
Domain Categorical (multiple select) The machine learning domain(s) of application associated with the model. This is fairly high-level, for example “Language” incorporates many different ML tasks.

Possible values: 3D modeling, Audio, Biology, Driving, Earth science, Games, Image generation, Language, Mathematics, Medicine, Multimodal, Other, Recommendation, Robotics, Search, Speech, Video, Vision
Language 100%
824 out of 824 models
Epochs Numeric How many epochs (repetitions of the training dataset) was used to train the model. 1 21%
169 out of 824 models
Finetune compute Numeric Compute used to fine-tune the model, if applicable. [empty]

This is empty because Llama-2 was not finetuned from a base model. For a non-empty example, consider CodeLlama, a Llama-2 finetune. The finetune compute would be the compute used for finetuning to train CodeLlama.
3%
28 out of 824 models
Hardware quantity Numeric Indicates the quantity of the hardware used in training, i.e. the number of chips. 1000 17%
141 out of 824 models
Hardware utilization Numeric Number between 0.00 and 1.00 indicating the hardware utilization ratio, i.e. utilized FLOPs / theoretical maximum FLOPs.

Where available, record Model FLOP Utilization, which does not depend on implementation details such as checkpointing. Almost all entries record this. However, when Hardware FLOP Utilization is the only reported value, this can be recorded instead.
0.4191975017 4%
29 out of 824 models
Link URL Link(s) to best-choice sources documenting a model. This should preferentially be a journal or conference paper, preprint, or technical report. If these are not available, the links should point to other supporting evidence, such as an announcement post, a news article, or similar. https://ai.meta.com/research/publications/llama-2-open-foundation-and-fine-tuned-chat-models/ https://arxiv.org/abs/2307.09288 100%
822 out of 824 models
Notability criteria Categorical (multiple select)

The criteria met by the model which qualify it for notability and therefore inclusion in the database. To be notable, a model must meet at least one criterion.

Possible values are highly cited, large training cost, significant use, state of the art, or historical significance. These are discussed further in Inclusion.

Historical significance, Significant use, Highly cited, Training cost 100%
824 out of 824 models
Organization Categorical (multiple select)

Organization(s) who created the model.

Organizations may have multiple different names, but we aim to standardize organization names where they refer to the same organization. Therefore, organizations are periodically reviewed in Airtable and standardized to the most common name for them.

For example, “University of California, Berkeley” and “Berkeley” have been changed to “UC Berkeley”. Note that some organizations have similar names but genuinely are different organizations, for example Google Brain versus Google versus Google DeepMind.

Meta AI 99%
819 out of 824 models
Organization categorization Categorical (multiple select) Categorization of the organization(s), automatically populated from the Organization entry. Models are categorized as “Industry” if their authors are affiliated with private sector organizations, “Academia” if the authors are affiliated with universities or academic institutions, or “Industry - Academia Collaboration” when at least 30% of the authors are from each. Possible values: Industry, Research Collective, Academia, Industry - Academia Collaboration (Industry leaning), Industry - Academia Collaboration (Academia leaning), Non-profit Industry 99%
819 out of 824 models
Parameters Numeric

Number of learnable parameters in the model. For neural networks, these are the weights and biases. Further information is provided in Estimation.

7.0e10 65%
538 out of 824 models
Publication date Date The publication, announcement, or release date of the model, in YYYY-MM-DD format. If the year and month are known but the day is unknown, the day is filled in as YYYY-MM-15. If the year is known but the month and day are unknown, the month and day are filled in as YYYY-07-01. 2023-07-18 100%
824 out of 824 models
Reference Text The literature reference for the model, such as the title of the journal or conference paper, academic preprint, or technical report. Llama 2: Open Foundation and Fine-Tuned Chat Models 99%
815 out of 824 models
System Text The name of the model. This should be unique within the database, and should be the best-known name for a given model.

This column must be filled in, and is used as the primary key for indexing entries in the database.
Llama 2-70B 100%
824 out of 824 models
Training compute Numeric

Quantity of compute used to train the model, in FLOP. This is the total training compute for a given model, i.e. pretrain + finetune. It should be filled in here when directly reported, or calculated via GPU-hours or backpropagation gradient updates. Further guidance is provided in Estimation.

8.1e23 47%
384 out of 824 models
Training compute cost (2023 USD) Numeric

The training compute cost, estimated using the “amortized hardware capex plus energy” approach documented in our training cost methodology. Values are converted to 2023 US dollars.

1,099,604 19%
156 out of 824 models
Training dataset Categorical (multiple select) Standard datasets are often used, and can be selected as multiple choice options. If a custom, unreleased dataset is used, it is set as "Unspecified Unreleased". Where this entry is empty, it has not yet been entered for a given model.

Datasets are standardized to their most common name. For example, “MS COCO” and “Microsoft COCO” are standardized as “COCO”.
Llama 2 dataset 49%
401 out of 824 models
Training dataset size Numeric

Number of datapoints in the training dataset, in the unit specified for a given task, for example number of images in image classification, or number of words in language modeling. Further guidance is provided in Estimation. This counts the dataset size as used for training, so e.g. if a model is trained on a subset of a public dataset, this field reflects the size of that subset.

1.5e12 45%
372 out of 824 models
Training hardware Multiple Select
Categorical
Type of training hardware used. Entries are cross-referenced against Epoch AI’s database of ML training hardware. NVIDIA A100 SXM4 80 GB 29%
243 out of 824 models
Training time (hours) Numeric

Training time of the model, if reported. This refers to the time elapsed over the training run, not the number of GPU-hours. So for example, if a model were trained with 10 GPUs for 1 hour, the training time would be 1 hour.

1728 20%
165 out of 824 models
Notes fields, e.g. “Training compute notes” Text Metadata documenting the reasoning and/or evidence for a given column, e.g. training compute or dataset size. This is particularly important to note in cases where such information isn’t obvious. This field is unstructured text. Training compute notes:

"Pretraining utilized a cumulative 3.3M GPU hours of computation on hardware of type A100-80GB" of which 1720320 GPU hours were used to train the 70B model. 311.84 BF16 TFLOP/s * 1720320 hours * 0.40 utilization = 7.725e+23 FLOP. Alternatively: the model was trained for 1 epoch on 2 trillion tokens and has 70B parameters. C = 6ND = 6*70B*2T = 8.4e+23 FLOP.

Database Updates

This section provides more information about recurring processes in the database: adding new models, updating citation counts, and updating the hosted files by which the database can be accessed for analysis.


Adding new models

Entries are added to the database near-daily, including both newly-released models and older models newly identified as notable. Typically, most information that can easily be determined from public information is added at the time a model is entered in the database. However, it is common for some information to gradually be entered later. For example, a compute estimate might be omitted at first and only added after we devote further effort to calculating it.

If you would like to ask any questions about the database, or suggest a model that should be added, feel free to contact us at data@epochai.org.


Updating citation counts

When models are added to the database, citation counts are recorded for those with academic publications or preprints. At the beginning of each month, citation counts are automatically updated for publications listed in Semantic Scholar. Publications not listed in Semantic Scholar rely on manual entry of citation count.


Updating hosted files

Epoch AI’s database is hosted as a CSV that is synced with the database daily. The easiest way to load the data in scripts is using the CSV URL. If you need the most up-to-date version reflecting unsynced changes, a CSV can be manually generated from the table view on the website.

Estimation

Some fields within the database require estimation, because they are often not straightforwardly reported within papers or other sources. Here, we detail how estimation works for compute, model size, dataset size, and the metadata on estimate confidence.


Estimating compute

Training compute is one of the most important pieces of information in our database, as reflected in its usage across Epoch AI’s research and elsewhere. However, estimating compute can be challenging. Here we outline how compute estimation is performed in the notable models database.

Compute is measured in units of floating point operations (FLOP). For older models, sometimes the relevant operations were integer operations - in this case we report these instead. We do not apply any multiplier to adjust for operations potentially being more valuable under different tensor formats or precisions, for example FP16 versus FP32 or BF16. Some sources report compute in multiplications-and-addition operations, fused multiply-adds (FMAs), or similar. We treat one multadd/FMA as being equivalent to two FLOP to match typical reporting of chip performance.

For a given model in the database, training compute is provided as the total training compute, including pretraining, and including pretrained base models used as components. Finetuning compute is recorded in its associated column. Finetuning is distinguished by authors’ descriptions of the training as finetuning, or unambiguous use of a pretrained model in a distinct phase of training.

In the simplest case, training compute is directly reported in a paper, and we enter this figure into the database. When compute is not reported, we use two main methods to estimate it:

  1. Hardware details and usage.
  2. Counting the operations based on model architecture and data.

When there is enough information to count the operations, this is preferred in our dataset, because typically hardware-based estimates require assumptions about utilization, which may reduce the estimates’ accuracy.

Hardware details and usage

Hardware details and usage is a relatively straightforward way to estimate compute, when the necessary details are known:

  1. The usage in chip-time, e.g. “trained on a cluster of 128 TPUv3 instances for two days” means 256 chip-days = 128 chips × 2 days. Sometimes this is reported as separate chips and time used, other times this may be reported directly in chip-time. When it is not reported, we may create estimates from publicly-known information, comparison to typical training runs, etc.
  2. The type of hardware used, e.g. NVIDIA H100, TPUv3. Ideally, this is reported in the paper. Otherwise, for more speculative estimates, one may have to make assumptions based on institution and year, e.g. that Google would have used TPUs of the corresponding generation in that year.
  3. The type of number representation used, e.g. FP32, FP16, BF16. Ideally, this is reported in the paper. When not reported, it can often be guessed. For example, the number representation was typically FP32 for models trained before 2019.

Once these details are known, the corresponding peak FLOP/s performance by hardware and number representation can be found from hardware documentation, or from the tool below. Finally, utilization rates account for real training runs falling significantly short of peak performance due to memory bottlenecks, network latency, etc. Typical utilization rates for large distributed training runs are around 30-50%. When these are not reported, they are estimated by reference to comparable models from a similar time period.

Table 2: Worked example of estimating training compute from hardware details.
ImageGPT

Some training details are provided in the blogpost: “[..]iGPT-L was trained for roughly 2500 V100-days […]”

The number representation is not specified, but given this was trained by a major corporation in 2020, we assume the number format was FP16.

The V100 has 125 TFLOP/s tensor FP16 performance. Assuming a utilization of 0.3, this leads to the following compute estimate:

8.1e21 FLOP = 2500 V100-days × 125e12 FLOP/s × 0.3 utilization × 86.4e3 s/day

Counting the operations

Counting the number of operations is often useful for older research, where hardware and usage details might be unavailable. The above formula sets out a widely-applicable heuristic for training compute of dense models. This works by first estimating required FLOP for a forward pass, which is approximately twice the number of connections. This can be modified by sparsity such as Mixture-of-Experts: in this case, the heuristic should use the number of connections in the number of active experts.

The forward pass FLOP is then multiplied by three to account for backward passes, as the ratio between forward and backward passes is 1:2 for non-recurrent dense models. Finally, this is multiplied by the number of passes performed on the data - the number of training examples multiplied by the number of epochs the model was trained. For transformer-based language models, this formula is equivalent to the commonly-used heuristic: Compute = 6 × # of parameters × # of training examples × # of epochs.

Sometimes, the FLOP for a forward pass is reported directly in a paper. In this case, this value can be used directly instead of 2 × # of connections. Otherwise, the FLOP for a forward pass are evaluated by summing FLOP over the network’s layers. These are set out in Table 3.

Table 3: Common neural network layers and associated FLOP per token in a forward pass.
Layer Forward pass FLOP per token (approx)
Fully connected layer from N neurons to M neurons 2×N×M
CNN from a tensor of shape H×W×C with D filters of shape K×K×C, applied with stride S and padding P 2×H^2×W^2×C×D/S^2
Transpose CNN from a tensor of shape H×W×C with D filters of shape K×K×C, applied with stride S and padding P 2×D×H×W×C^2×K^2
RNN with bias vectors taking an input of size N and producing an output of size M 2×(N+M)×M
Fully gated GRU with bias vectors taking an input of size N and producing an output of size M 6×(N+M)×M
LSTM with bias vectors taking an input of size N and producing an output of size M 8×(N+M)×M
Word Embedding for vocabulary size V and embedding dimension W 0
Self attention layer with sequence length L, inputs of size W, key of size D and output of size N 2×W×(2×D+N) + 2×L×(D+N)
Multi-headed attention layer with sequence length L, inputs of size W, key of size D, head output of size N, output of size M and H attention heads 2×H×(W×(2×D+N) + L×(D+N) + N×M)
Table 4: Worked example of estimating training compute from architecture.
Attention Is All You Need

The input is a sequence of tokens, with an average length of 20 and a vocabulary size of 30,000. Each token is embedded and represented as a vector of size 1024. There are six encoder and decoder layers. Each encoder-decoder pair has a total of 3 multi-headed attention (MHA) sublayers, and 2 fully connected (FCN) sublayers. At the end there is a final linear layer and a softmax.

Each MHA sublayer has the following parameters: input size W=64, head output size N=64, key size D=64, number of heads H=16, final output size M=1024. Hence each MHA sublayer has 2×16×(64×(2×64+64) + 20×(64+64) + 64×1024) = 2.6e6 FLOP per token.

Each FCN sublayer has an input size of 1024, output size of 1024, and a single hidden layer with 4096 units. Hence each FCN sublayer has 2×2×1024×4096 = 1.7e7 FLOP per token.

Summing all its layers, the encoder-decoder stack has 6 × (3 × 2.6e6 + 2 × 1.7e7) ~= 2.5e8 FLOP per token. The final linear layer has 2 × 1024 × 3e4 = 6.1e7 FLOP per token. Summing these, a forward pass takes 3.1e8 FLOP per token.

The paper says they use batches of 25,000 tokens, and run the training for 300,000 steps. So the total training FLOP would be 2.5e4 × 3e5 × 3 × 3.1e8 = 6.97e18 FLOP.


Estimating model size

Parameter counts are often reported by the model developer, but if parameter count is not stated, it can sometimes be estimated based on provided architectural details. Similar to estimating compute, estimating parameter count requires finding a description of the architecture, i.e. type, number, and configuration of the layers, then calculating the parameters in each layer and summing them. Table 5 lists the parameter counts for different layers. Alternatively, if an architecture implementation is available in code, it can be simpler to load an architecture in code and count the number of parameters.

Table 5: Common neural network layers and parameters.
Layer Parameters (approx)
Fully connected layer from N neurons to M neurons N×M
CNN from a tensor of shape H×W×C with D filters of shape K×K×C, applied with stride S and padding P D×K^2×C
Transpose CNN from a tensor of shape H×W×C with D filters of shape K×K×C, applied with stride S and padding P D×K^2×C
RNN with bias vectors taking an input of size N and producing an output of size M (N+M)×M
Fully gated GRU with bias vectors taking an input of size N and producing an output of size M 3×(N+M)×M
LSTM with bias vectors taking an input of size N and producing an output of size M 4×(N+M)×M
Word Embedding for vocabulary size V and embedding dimension W W×V
Self attention layer with sequence length L, inputs of size W, key of size D and output of size N W×(2×D+N)
Multi-headed attention layer with sequence length L, inputs of size W, key of size D, head output of size N, output of size M and H attention heads H×(W×(2×D + N) + N×M)
Table 6: Worked example of estimating training model size from architecture.
Attention Is All You Need

The input is a sequence of tokens, with an average length of 20 and a vocabulary size of 30,000. Each token is embedded and represented as a vector of size 1024. There are six encoder and decoder layers. Each encoder-decoder pair has a total of 3 multi-headed attention (MHA) sublayers, and 2 fully connected (FCN) sublayers. At the end there is a final linear layer and a softmax.

Each MHA sublayer has the following parameters: input size W=64, head output size N=64, key size D=64, number of heads H=16, final output size M=1024. Hence each MHA sublayer has 16×(64×(2×64 + 64) + 64×1024) = 1.2e6 parameters.

Each FCN layer has an input size of 1024, output size of 1024, and a single hidden layer with 4096 units. Hence each FCN layer has 2×1024×4096 = 8.4e6 parameters.

Summing all its layers, the encoder-decoder stack has 6 × (3 × 1.2e6 + 2 × 8.4e6) ~= 1.2e8 parameters. The final linear layer has 1024 × 3e4 = 3.1e7 parameters. Two embedding layers each have 30e3 × 1024 parameters, so 6.2e7 in total. Summing these, the model has 2.1e8 parameters, matching the reported 213 million parameters in the paper.


Estimating dataset size

Dataset size is measured in the number of datapoints used as training examples, as outlined in Table 7. The objective here is to provide an intuitive idea of how large a dataset is. We record the number of distinct datapoints, not multiplied by the number of epochs. The number of epochs is recorded separately. Table 8 provides worked examples across several of these ML problems.

Table 7: How to measure dataset size for different ML problems.
ML problem Way of measuring dataset size
Classification # training examples
Image classification # images
Image captioning # captions
Language modeling # words
Translation # words in input language
Text classification # training examples
Speech recognition # words
Reinforcement learning # timesteps
Table 8: Worked examples for calculating dataset size across different ML problems.
Image classification: Deep Residual Learning for Image Recognition

“We evaluate our method on the ImageNet 2012 classification dataset that consists of 1000 classes. The models are trained on the 1.28 million training images, and evaluated on the 50k validation images.”

We thus note down a dataset size of 1.28e6 (the number of images).

Image captioning: Deep Residual Learning for Image Recognition

“We evaluate our method on the ImageNet 2012 classification dataset that consists of 1000 classes. The models are trained on the 1.28 million training images, and evaluated on the 50k validation images.”

According to the authors, the MSCOCO dataset is “arguably the largest and highest quality dataset” that they used. This had 82,783 training examples, each containing a single image and 5 sentences that are “relatively visual and unbiased”. To determine the dataset size, we consider the number of image-caption pairs. Thus we count 82,783 * 5 = 413,915 training examples.

Language modeling: Language Models are Few-Shot Learners

From the paper, we determine that there are 410 + 19 + 12 + 55 + 3 = 499 billion tokens.

We convert this to words by multiplying by 0.75 to give 374B words.

Speech: An RNN-based prosodic information synthesizer for Mandarin text-to-speech

“A continuous-speech Mandarin database provided by the Telecommunication Laboratories, MOTC,1 R.O.C. was used… The data base was divided into two parts: a training set and an open test set. These two sets consisted of 28191 and 7051 syllables, respectively.”

We convert this to words by multiplying 28,191 syllables by 0.62 to get 17,478 words.

Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. These factors are based on the OpenAI GPT-3 tokenizer for Western languages, and manual inspection that tokenizers typically have one token per word in Mandarin, Japanese and Korean. The ratio is tokenizer-dependent, meaning that when estimating a dataset’s size in words, one should consider whether the tokenizer might have a substantially different ratio. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10.

Table 9: Conversion between words, tokens, and GB for different languages.
Language Words per token Words per GB (approx)
English 0.75 200M
Mandarin Chinese 1 167M
German 0.75 167M
Spanish 0.75 200M
Japanese 1 200M
Korean 1 111M
Table 10: Conversion between words, minutes/hours, and syllables for different languages. Adapted from Trauzettel-Klosinksi et al. (2012).
Language Words per minuteWPM Words per hourWPH Words per syllableWPS
English 228 13,680 ~0.73
Mandarin Chinese 158 9,480 ~0.62
German 179 10,740 ~0.59
Spanish 218 13,080 ~0.41
Japanese 193 11,580 ~0.43

Estimating confidence

As discussed in Records, the confidence statuses specify the following bounds as 90% confidence intervals:

  • Confident - ±3x (~0.5 orders of magnitude).
  • Likely - ±10x (1 order of magnitude).
  • Speculative - ±31x ( ~1.5 orders of magnitude).

Confidence applies to the recorded values for Training compute, Parameters, and Training dataset size. It describes confidence for the most uncertain of these values, for the ones that have a non-empty entry.

To estimate confidence statuses, we consider which parts of an estimate are uncertain, and how large the uncertainty is.

  • If details (compute, model size, and dataset size) are all directly reported, then the value is Confident. There is little room for error.
  • If a detail is estimated without any assumptions having to be made, then the value is Confident. For example, if hardware type, quantity, training time, number format, and utilization are all reported, then the ensuing compute estimate is unambiguous.
  • When details are ambiguous, and an assumption has to be made, we consider the uncertainty in that assumption.
  • For example, it is often necessary to estimate utilization when estimating training compute from hardware details. Given the typical language modeling range is 0.3-0.5, this estimate should fall within the Confident category.
  • Further ambiguity may move estimates into the Likely category. For example, MedBERT was trained for one week using one V100 GPU, but the authors do not report the arithmetic precision or usage of tensor cores during training, which could affect the compute usage by a factor of 4x.
  • Finally, some estimates are based almost entirely on credible ranges for (unreported) key parameters such as training time and hardware. These typically fall into the Speculative category. An example of this is GPT-4, where our compute estimate is based on secondhand reporting that lets us roughly estimate training duration and hardware.

Changelog

2024-06-19

The documentation was updated for the launch of the dataset on Epoch AI’s “Data on AI” webpage.

  • Updates particularly affected sections on estimating compute, parameters, and dataset sizes
  • The confidence field was updated to be defined in terms of 90% confidence intervals for estimated values.
  • The documentation was restructured for clarity. 

Downloads

Notable Machine Learning Models

CSV, Updated July 23, 2024

In addition to the notable models database, we also host an “all models” database with entries covering additional models used in our other research projects. Many of these models do not qualify as notable under our inclusion criteria. We do not recommend using this expanded database unless you have a specific reason to do so, for example because less notable models are of interest in your research.

Acknowledgements

We would like to thank the authors of several sources where we have found one or more ML models to include in the database: Stanford CRFM’s foundation model ecosystem graph, AI Tracker, Stella Biderman’s directory of LLMs, Terry Um’s repo of deep learning papers, Alan Thompson’s models table, the OpenCompass Chinese LM leaderboard, the Akronomikon by LightOn AI, Papers With Code, the Metaculus 2022 AI Forecasting Database, and Hugging Face. We would also like to thank the authors of AI and compute and Compute and Energy Consumption Trends in Deep Learning Inference.

The data have been collected by Epoch AI’s employees and collaborators, including Jaime Sevilla, Pablo Villalobos, Juan Felipe Cerón, Matthew Burtell, Lennart Heim, Amogh B. Nanjajjar, Tilman Rauker, Nuño Sempere, Max Rauker, Anson Ho, Tamay Besiroglu, Marius Hobbhahn, Jean-Stanislas Denain, Owen Dudney, David Atkinson, Ben Cottier, David Owen, Robi Rahman, Carl Guo, Josh You, Nicole Maug, Aidan O’Gara, Bartosz Podkanowicz, Luke Frymire, and Natalia Martemianova.

This documentation was written by David Owen and Robi Rahman. Material on estimating compute, parameters and dataset sizes was adapted from previous documents by Jaime Sevilla, Lennart Heim, Marius Hobbhahn, Tamay Besiroglu, Anson Ho, Pablo Villalobos, and Robi Rahman.