Deep Learning Scaling is Predictable, Empirically

1 December 2017

Sharan Narang

Md. Mostofa Ali Patwary

Yang Yang

Yanqi Zhou

ArXiv (abs)PDF HTML

Papers citing "Deep Learning Scaling is Predictable, Empirically"

50 / 372 papers shown

Title
Integration of Convolutional Neural Networks in Mobile Applications Roger Creus Castanyer Silverio Martínez-Fernández Xavier Franch 59 12 0 11 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 1.1K 30,111 0 26 Feb 2021
Efficient Client Contribution Evaluation for Horizontal Federated Learning Jie Zhao Xinghua Zhu Jianzong Wang Jing Xiao FedML 95 28 0 26 Feb 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 104 272 0 12 Feb 2021
Learning Curve Theory Marcus Hutter 231 64 0 08 Feb 2021
An Update on a Progressively Expanded Database for Automated Lung Sound Analysis Fu-Shun Hsu Shang-Ran Huang Chien-Wen Huang Yuan-Ren Cheng Chun-Chieh Chen Jack Hsiao Chung-Wei Chen F. Lai 44 7 0 08 Feb 2021
Network Support for High-performance Distributed Machine Learning F. Malandrino C. Chiasserini Nuria Molner Antonio de la Oliva 92 10 0 05 Feb 2021
Benchmarking of eight recurrent neural network variants for breath phase and adventitious sound detection on a self-developed open-access lung sound database-HF_Lung_V1 Fu-Shun Hsu Shang-Ran Huang Chien-Wen Huang Chao-Jung Huang Yuan-Ren Cheng ... Yi-Lin Wu Tzu-Ling Tzeng Ching-Ting Tseng Yi-Tsun Chen F. Lai 81 57 0 05 Feb 2021
E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials Simon L. Batzner Albert Musaelian Lixin Sun Mario Geiger J. Mailoa M. Kornbluth N. Molinari Tess E. Smidt Boris Kozinsky 324 1,340 0 08 Jan 2021
A Clinical Evaluation of a Low-Cost Strain Gauge Respiration Belt and Machine Learning to Detect Sleep Apnea Stein Kristiansen K. Nikolaidis T. Plagemann V. Goebel G. Traaen ... S. Steinshamn C. Bendz O. Anfinsen L. Gullestad Harriet Akre 27 15 0 07 Jan 2021
Analysis of the Scalability of a Deep-Learning Network for Steganography "Into the Wild" Hugo Ruiz Marc Chaumont Mehdi Yedroudj A. Amara Frédéric Comby Gérard Subsol 80 9 0 29 Dec 2020
*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task Dmitry Tsarkov Tibor Tihon Nathan Scales Nikola Momchev Danila Sinopalnikov Nathanael Scharli 76 17 0 15 Dec 2020
Generalization bounds for deep learning Guillermo Valle Pérez A. Louis BDL 84 45 0 07 Dec 2020
Learning Curves for Drug Response Prediction in Cancer Cell Lines A. Partin Thomas Brettin Yvonne A. Evrard Yitan Zhu H. Yoo ... Austin R. Clyde Maulik Shukla Michael Fonstein J. Doroshow Rick L. Stevens 68 20 0 25 Nov 2020
Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues A. Alam I. Ullah Young-Koo Lee 70 25 0 16 Nov 2020
Video Big Data Analytics in the Cloud: Research Issues and Challenges A. Alam S. Khalid Muhammad Numan Khan Tariq Habib Afridi I. Ullah Young-Koo Lee 112 1 0 05 Nov 2020
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference Michael Lui Yavuz Yetim Özgür Özkan Zhuoran Zhao Shin-Yeh Tsai Carole-Jean Wu Mark Hempstead GNN BDL LRM 79 52 0 04 Nov 2020
CopyPaste: An Augmentation Method for Speech Emotion Recognition R. Pappagari Jesús Villalba Piotr Żelasko Laureano Moro-Velazquez Najim Dehak 73 41 0 27 Oct 2020
Are wider nets better given the same number of parameters? A. Golubeva Behnam Neyshabur Guy Gur-Ari 112 44 0 27 Oct 2020
The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research N. Ahmed Muntasir Wahed 87 111 0 22 Oct 2020
Deep Learning is Singular, and That's Good Daniel Murfet Susan Wei Biwei Huang Hui Li Jesse Gell-Redman T. Quella UQCV 79 29 0 22 Oct 2020
Transferable Graph Optimizers for ML Compilers Yanqi Zhou Sudip Roy AmirAli Abdolrashidi Daniel Wong Peter C. Ma ... Mangpo Phitchaya Phothilimtha Shen Wang Anna Goldie Azalia Mirhoseini James Laudon GNN 73 55 0 21 Oct 2020
Small Data, Big Decisions: Model Selection in the Small-Data Regime J. Bornschein Francesco Visin Simon Osindero 65 40 0 26 Sep 2020
Pruning Convolutional Filters using Batch Bridgeout Najeeb Khan Ian Stavness 42 3 0 23 Sep 2020
Action-Based Representation Learning for Autonomous Driving Yi Xiao Felipe Codevilla C. Pal Antonio M. López 86 10 0 21 Aug 2020
Geometric compression of invariant manifolds in neural nets J. Paccolat Leonardo Petrini Mario Geiger Kevin Tyloo Matthieu Wyart MLT 105 36 0 22 Jul 2020
Add a SideNet to your MainNet Adrien Morisot 23 0 0 14 Jul 2020
The Computational Limits of Deep Learning Neil C. Thompson Kristjan Greenewald Keeheon Lee Gabriel F. Manso VLM 101 533 0 10 Jul 2020
Is SGD a Bayesian sampler? Well, almost Chris Mingard Guillermo Valle Pérez Joar Skalse A. Louis BDL 81 53 0 26 Jun 2020
On the Predictability of Pruning Across Scales Jonathan S. Rosenfeld Jonathan Frankle Michael Carbin Nir Shavit 82 38 0 18 Jun 2020
To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks Sinong Wang Madian Khabsa Hao Ma 60 26 0 15 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 1.2K 42,749 0 28 May 2020
How fine can fine-tuning be? Learning efficient language models Evani Radiya-Dixit Xin Wang 53 66 0 24 Apr 2020
Embedded Large-Scale Handwritten Chinese Character Recognition Youssouf Chherawala Hans J. G. A. Dolfing Ryan S. Dixon J. Bellegarda 14 5 0 13 Apr 2020
SuperMix: Supervising the Mixing Data Augmentation Ali Dabouei Sobhan Soleymani Fariborz Taherkhani Nasser M. Nasrabadi 128 101 0 10 Mar 2020
Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models Ki Hyun Tae Steven Euijong Whang 85 41 0 10 Mar 2020
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar Cengiz Pehlevan 286 208 0 07 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 673 4,945 0 23 Jan 2020
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence Kihyuk Sohn David Berthelot Chun-Liang Li Zizhao Zhang Nicholas Carlini E. D. Cubuk Alexey Kurakin Han Zhang Colin Raffel AAML 173 3,603 0 21 Jan 2020
Designing for the Long Tail of Machine Learning Martin Lindvall J. Molin HAI 13 2 0 21 Jan 2020
Value-laden Disciplinary Shifts in Machine Learning Ravit Dotan S. Milli AILaw 84 48 0 03 Dec 2019
Using Error Decay Prediction to Overcome Practical Issues of Deep Active Learning for Named Entity Recognition Haw-Shiuan Chang Shankar Vembu Sunil Mohan Rheeya Uppaal Andrew McCallum 43 3 0 17 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 915 20,461 0 23 Oct 2019
Improving Differentially Private Models with Active Learning Zhengli Zhao Nicolas Papernot Sameer Singh N. Polyzotis Augustus Odena SyDa 38 5 0 02 Oct 2019
GDP: Generalized Device Placement for Dataflow Graphs Yanqi Zhou Sudip Roy AmirAli Abdolrashidi Daniel Wong Peter C. Ma ... Ming Zhong Hanxiao Liu Anna Goldie Azalia Mirhoseini James Laudon GNN 77 39 0 28 Sep 2019
A Constructive Prediction of the Generalization Error Across Scales Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit 113 215 0 27 Sep 2019
Data Valuation using Reinforcement Learning Jinsung Yoon Sercan O. Arik Tomas Pfister TDI 93 183 0 25 Sep 2019
Beyond Human-Level Accuracy: Computational Challenges in Deep Learning Joel Hestness Newsha Ardalani G. Diamos 64 68 0 03 Sep 2019
P2L: Predicting Transfer Learning for Images and Semantic Relations Bishwaranjan Bhattacharjee J. Kender Matthew Q. Hill Parijat Dube Siyu Huo Michael R. Glass Brian M. Belgodere Sharath Pankanti Noel Codella Patrick Watson VLM 83 13 0 20 Aug 2019
TabNet: Attentive Interpretable Tabular Learning Sercan O. Arik Tomas Pfister LMTD 234 1,390 0 20 Aug 2019