Notable AI Models Documentation

Inclusion

The database focuses on notable ML models: models that have advanced the state of the art, had a large influence in the field’s history, or had a large impact within the world. Here, we detail criteria for inclusion, and give an overview of how the data have been collected.

---

Criteria

To be included in the database, an ML model must satisfy all inclusion criteria:

• there must be reliable documentation of its existence and relevance to machine learning;
• the model must include a learning component, it cannot be a non-learned algorithm;
• the model must actually have been trained, it cannot be a theoretical description without experimental results;
• the model must be notable, per any of the notability criteria defined below.

Notability

Models are notable if they satisfy any of the following:

• highly cited (over 1000 citations);
• large training cost (over $5,000,000, measured in 2023 USD);
• significant use (over one million monthly active users);
• state of the art performance (typically on a recognised ML benchmark, see below for further discussion);
• indisputable historical significance.

Where there are many related models, for example several checkpoints along training or several sizes of a given model family, the database preferentially includes the version that used the most compute. Other versions may be included where they are notable in their own right.

State of the art

Identifying whether a model is state of the art can be a more involved process, compared to simply checking citations or the training compute budget. We consider a model to be state of the art if there is good reason to believe that it was the best existing model at the time for a task of genuine interest. The default way to provide evidence for this is state of the art performance on a recognised benchmark.

To be recognised, a benchmark should have any of the following:

• 100+ citations.
• 10+ submissions in total from 3+ research groups.
• An associated publication in a reputable peer-reviewed academic venue. The publication does not need to focus exclusively on the benchmark; however, the benchmark should be a key result.

At our discretion, we may also identify models as state of the art where no benchmark result exists, but there is convincing evidence that a model truly is state of the art. Eligible sources of evidence here are comparison on a non-benchmark database, a high-quality user preference study, or demonstration of state of the art capabilities. For example, GraphCast is compared against other weather prediction models on a weather database that is not a standalone benchmark. Nevertheless, we take this as convincing evidence that it is state of the art.

Historical significance

Models can be included on the grounds of historical significance if they marked a significant advance in AI history, even if they did not strictly advance the state of the art on any application. For example, many neural network breakthroughs performed worse than other ML techniques, but were directly influential for later AI development. Evidence to support this status may come from citations in later notable models, discussion in reviews or textbooks, or other unambiguous identification as an influential result.

---

Search process

This database has been collected from a variety of sources: literature reviews, historical accounts of AI development, highly-cited publications from top conferences, high-profile models from leading industry labs, bibliographies of notable papers, pre-existing datasets curating AI papers (see Acknowledgements), and ad hoc suggestions from contributors.

We monitor news coverage, releases from key AI labs, and benchmarks to identify new notable models as they are released. This can lead to a lag for new models. Typically, we aim to add the most prominent releases (e.g. GPT-4) within days of release. For less prominent models, reporting lags may extend to months.

---

Coverage

As of June 28, 2024, the dataset contains 825 models, 387 of which have compute estimates.

The database does not provide exhaustive coverage of notable models. However, data collection efforts to support Epoch AI research projects have led to more thorough coverage within particular niches, such as models trained with large-scale compute, and biological sequence models.

• Coverage is most thorough for language and vision models developed since 2018 (257 models and 111 models respectively), albeit with a lag for the newest models. More specialist domains, such as robotics, likely have worse coverage in this period.
• Coverage is fair, but less thorough, for deep learning language and vision models between 2010-2018 (72 models and 100 models respectively). Again, other domains may have worse coverage.
• Coverage is quite sparse for historical models before 2010 (143 models before 2010 compared to 682 models after), particularly models outside the paradigm of deep learning. Entries here are focused on notable models mentioned in textbooks and reviews, rather than a systematic search across sources.

If you would like to ask any questions about the database, or suggest a model that should be added, feel free to contact us at data@epochai.org.

Database Updates

This section provides more information about recurring processes in the database: adding new models, updating citation counts, and updating the hosted files by which the database can be accessed for analysis.

---

Adding new models

Entries are added to the database near-daily, including both newly-released models and older models newly identified as notable. Typically, most information that can easily be determined from public information is added at the time a model is entered in the database. However, it is common for some information to gradually be entered later. For example, a compute estimate might be omitted at first and only added after we devote further effort to calculating it.

If you would like to ask any questions about the database, or suggest a model that should be added, feel free to contact us at data@epochai.org.

---

Updating citation counts

When models are added to the database, citation counts are recorded for those with academic publications or preprints. At the beginning of each month, citation counts are automatically updated for publications listed in Semantic Scholar. Publications not listed in Semantic Scholar rely on manual entry of citation count.

---

Updating hosted files

Epoch AI’s database is hosted as a CSV that is synced with the database daily. The easiest way to load the data in scripts is using the CSV URL. If you need the most up-to-date version reflecting unsynced changes, a CSV can be manually generated from the table view on the website.

Estimation

Some fields within the database require estimation, because they are often not straightforwardly reported within papers or other sources. Here, we detail how estimation works for compute, model size, dataset size, and the metadata on estimate confidence.

---

Estimating compute

Training compute is one of the most important pieces of information in our database, as reflected in its usage across Epoch AI’s research and elsewhere. However, estimating compute can be challenging. Here we outline how compute estimation is performed in the notable models database.

Compute is measured in units of floating point operations (FLOP). For older models, sometimes the relevant operations were integer operations - in this case we report these instead. We do not apply any multiplier to adjust for operations potentially being more valuable under different tensor formats or precisions, for example FP16 versus FP32 or BF16. Some sources report compute in multiplications-and-addition operations, fused multiply-adds (FMAs), or similar. We treat one multadd/FMA as being equivalent to two FLOP to match typical reporting of chip performance.

For a given model in the database, training compute is provided as the total training compute, including pretraining, and including pretrained base models used as components. Finetuning compute is recorded in its associated column. Finetuning is distinguished by authors’ descriptions of the training as finetuning, or unambiguous use of a pretrained model in a distinct phase of training.

In the simplest case, training compute is directly reported in a paper, and we enter this figure into the database. When compute is not reported, we use two main methods to estimate it:

1. Hardware details and usage.
2. Counting the operations based on model architecture and data.

When there is enough information to count the operations, this is preferred in our dataset, because typically hardware-based estimates require assumptions about utilization, which may reduce the estimates’ accuracy.

Hardware details and usage

Hardware details and usage is a relatively straightforward way to estimate compute, when the necessary details are known:

1. The usage in chip-time, e.g. “trained on a cluster of 128 TPUv3 instances for two days” means 256 chip-days = 128 chips × 2 days. Sometimes this is reported as separate chips and time used, other times this may be reported directly in chip-time. When it is not reported, we may create estimates from publicly-known information, comparison to typical training runs, etc.
2. The type of hardware used, e.g. NVIDIA H100, TPUv3. Ideally, this is reported in the paper. Otherwise, for more speculative estimates, one may have to make assumptions based on institution and year, e.g. that Google would have used TPUs of the corresponding generation in that year.
3. The type of number representation used, e.g. FP32, FP16, BF16. Ideally, this is reported in the paper. When not reported, it can often be guessed. For example, the number representation was typically FP32 for models trained before 2019.

Once these details are known, the corresponding peak FLOP/s performance by hardware and number representation can be found from hardware documentation, or from the tool below. Finally, utilization rates account for real training runs falling significantly short of peak performance due to memory bottlenecks, network latency, etc. Typical utilization rates for large distributed training runs are around 30-50%. When these are not reported, they are estimated by reference to comparable models from a similar time period.

Counting the operations

Counting the number of operations is often useful for older research, where hardware and usage details might be unavailable. The above formula sets out a widely-applicable heuristic for training compute of dense models. This works by first estimating required FLOP for a forward pass, which is approximately twice the number of connections. This can be modified by sparsity such as Mixture-of-Experts: in this case, the heuristic should use the number of connections in the number of active experts.

The forward pass FLOP is then multiplied by three to account for backward passes, as the ratio between forward and backward passes is 1:2 for non-recurrent dense models. Finally, this is multiplied by the number of passes performed on the data - the number of training examples multiplied by the number of epochs the model was trained. For transformer-based language models, this formula is equivalent to the commonly-used heuristic: Compute = 6 × # of parameters × # of training examples × # of epochs.

Sometimes, the FLOP for a forward pass is reported directly in a paper. In this case, this value can be used directly instead of 2 × # of connections. Otherwise, the FLOP for a forward pass are evaluated by summing FLOP over the network’s layers. These are set out in Table 3.

---

Estimating model size

Parameter counts are often reported by the model developer, but if parameter count is not stated, it can sometimes be estimated based on provided architectural details. Similar to estimating compute, estimating parameter count requires finding a description of the architecture, i.e. type, number, and configuration of the layers, then calculating the parameters in each layer and summing them. Table 5 lists the parameter counts for different layers. Alternatively, if an architecture implementation is available in code, it can be simpler to load an architecture in code and count the number of parameters.

---

Estimating dataset size

Dataset size is measured in the number of datapoints used as training examples, as outlined in Table 7. The objective here is to provide an intuitive idea of how large a dataset is. We record the number of distinct datapoints, not multiplied by the number of epochs. The number of epochs is recorded separately. Table 8 provides worked examples across several of these ML problems.

Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. Language dataset sizes are usually reported in terms of tokens or gigabytes. These are converted to words per Table 9. These factors are based on the OpenAI GPT-3 tokenizer for Western languages, and manual inspection that tokenizers typically have one token per word in Mandarin, Japanese and Korean. The ratio is tokenizer-dependent, meaning that when estimating a dataset’s size in words, one should consider whether the tokenizer might have a substantially different ratio. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10. Speech recognition data are often expressed in terms of duration or syllables, which are converted to words per Table 10.

---

Estimating confidence

As discussed in Records, the confidence statuses specify the following bounds as 90% confidence intervals:

• Confident - ±3x (~0.5 orders of magnitude).
• Likely - ±10x (1 order of magnitude).
• Speculative - ±31x ( ~1.5 orders of magnitude).

Confidence applies to the recorded values for Training compute, Parameters, and Training dataset size. It describes confidence for the most uncertain of these values, for the ones that have a non-empty entry.

To estimate confidence statuses, we consider which parts of an estimate are uncertain, and how large the uncertainty is.

• If details (compute, model size, and dataset size) are all directly reported, then the value is Confident. There is little room for error.
• If a detail is estimated without any assumptions having to be made, then the value is Confident. For example, if hardware type, quantity, training time, number format, and utilization are all reported, then the ensuing compute estimate is unambiguous.
• When details are ambiguous, and an assumption has to be made, we consider the uncertainty in that assumption.
• For example, it is often necessary to estimate utilization when estimating training compute from hardware details. Given the typical language modeling range is 0.3-0.5, this estimate should fall within the Confident category.
• Further ambiguity may move estimates into the Likely category. For example, MedBERT was trained for one week using one V100 GPU, but the authors do not report the arithmetic precision or usage of tensor cores during training, which could affect the compute usage by a factor of 4x.
• Finally, some estimates are based almost entirely on credible ranges for (unreported) key parameters such as training time and hardware. These typically fall into the Speculative category. An example of this is GPT-4, where our compute estimate is based on secondhand reporting that lets us roughly estimate training duration and hardware.