Producing datasets and analysing trends in Machine Learning

Despite surging interest in Machine Learning, there has been limited work systematically curating and studying datasets about what these systems are like—how much compute they used, what datasets they were trained on, and what their architectures are like.

This work will help us build a big picture understanding of what has happened in the field in the recent decades, and ultimately help us understand where it might go next.

This line of research involves:

• Developing standards for collecting and representing data on Machine Learning systems
• Building datasets and making these publicly available for other researchers to use
• Creating measuring tools to estimate or extract features of ML systems, such as compute used during training
• Analysing and explaining trends in the data, investigating discontinuities, and plausible contributing factors
• Analysing the implications of a continuation of existing trends, by, for example, producing extrapolations and projections

Prior work