Compute Trends Across Three Eras of Machine Learning

Compute Trends Across Three Eras of Machine Learning

11 February 2022

Marius Hobbhahn

Pablo Villalobos

Papers citing "Compute Trends Across Three Eras of Machine Learning"

17 / 117 papers shown

Title
Machine Learning Model Sizes and the Parameter Gap Pablo Villalobos J. Sevilla T. Besiroglu Lennart Heim A. Ho Marius Hobbhahn ALM ELM AI4CE 30 58 0 05 Jul 2022
Researching Alignment Research: Unsupervised Analysis Jan H. Kirchner Logan Smith Jacques Thibodeau Kyle McDonell Laria Reynolds 28 6 0 06 Jun 2022
Towards Climate Awareness in NLP Research Daniel Hershcovich Nicolas Webersinke Mathias Kraus J. Bingler Markus Leippold 35 32 0 10 May 2022
Experimental Standards for Deep Learning in Natural Language Processing Research Dennis Ulmer Elisa Bassignana Max Müller-Eberstein Daniel Varab Mike Zhang Rob van der Goot Christian Hardmeier Barbara Plank 19 10 0 13 Apr 2022
Federated Self-supervised Speech Representations: Are We There Yet? Yan Gao Javier Fernandez-Marques Titouan Parcollet Abhinav Mehrotra Nicholas D. Lane 35 13 0 06 Apr 2022
Carbon Footprint of Selecting and Training Deep Learning Models for Medical Image Analysis Raghavendra Selvan N. Bhagwat Lasse F. Wolff Anthony Benjamin Kanding Erik Dam 36 24 0 04 Mar 2022
Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Sanjith Athlur Nitika Saran Muthian Sivathanu Ramachandran Ramjee Nipun Kwatra GNN 31 80 0 07 Nov 2021
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining Junyang Lin An Yang Jinze Bai Chang Zhou Le Jiang ... Jie Zhang Yong Li Wei Lin Jingren Zhou Hongxia Yang MoE 92 43 0 08 Oct 2021
Primer: Searching for Efficient Transformers for Language Modeling David R. So Wojciech Mañke Hanxiao Liu Zihang Dai Noam M. Shazeer Quoc V. Le VLM 88 152 0 17 Sep 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 316 3,708 0 11 Feb 2021
The Computational Limits of Deep Learning Neil C. Thompson Kristjan Greenewald Keeheon Lee Gabriel F. Manso VLM 23 506 0 10 Jul 2020
Meta Pseudo Labels Hieu H. Pham Zihang Dai Qizhe Xie Minh-Thang Luong Quoc V. Le VLM 262 656 0 23 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 271 5,329 0 05 Nov 2016
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,746 0 26 Sep 2016
Improving neural networks by preventing co-adaptation of feature detectors Geoffrey E. Hinton Nitish Srivastava A. Krizhevsky Ilya Sutskever Ruslan Salakhutdinov VLM 266 7,636 0 03 Jul 2012