Deep Learning Scaling is Predictable, Empirically

1 December 2017

Sharan Narang

Md. Mostofa Ali Patwary

Yang Yang

Yanqi Zhou

ArXiv PDF HTML

Papers citing "Deep Learning Scaling is Predictable, Empirically"

50 / 386 papers shown

Title
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 183 27,846 0 26 Feb 2021
Efficient Client Contribution Evaluation for Horizontal Federated Learning Jie Zhao Xinghua Zhu Jianzong Wang Jing Xiao FedML 36 28 0 26 Feb 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 27 250 0 12 Feb 2021
Learning Curve Theory Marcus Hutter 140 59 0 08 Feb 2021
An Update on a Progressively Expanded Database for Automated Lung Sound Analysis Fu-Shun Hsu Shang-Ran Huang Chien-Wen Huang Yuan-Ren Cheng Chun-Chieh Chen Jack Hsiao Chung-Wei Chen F. Lai 13 7 0 08 Feb 2021
Network Support for High-performance Distributed Machine Learning F. Malandrino C. Chiasserini Nuria Molner Antonio de la Oliva 52 10 0 05 Feb 2021
Benchmarking of eight recurrent neural network variants for breath phase and adventitious sound detection on a self-developed open-access lung sound database-HF_Lung_V1 Fu-Shun Hsu Shang-Ran Huang Chien-Wen Huang Chao-Jung Huang Yuan-Ren Cheng ... Yi-Lin Wu Tzu-Ling Tzeng Ching-Ting Tseng Yi-Tsun Chen F. Lai 43 53 0 05 Feb 2021
E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentials Simon L. Batzner Albert Musaelian Lixin Sun Mario Geiger J. Mailoa M. Kornbluth N. Molinari Tess E. Smidt Boris Kozinsky 233 1,240 0 08 Jan 2021
A Clinical Evaluation of a Low-Cost Strain Gauge Respiration Belt and Machine Learning to Detect Sleep Apnea Stein Kristiansen K. Nikolaidis T. Plagemann V. Goebel G. Traaen ... S. Steinshamn C. Bendz O. Anfinsen L. Gullestad Harriet Akre 16 14 0 07 Jan 2021
Perspective: A Phase Diagram for Deep Learning unifying Jamming, Feature Learning and Lazy Training Mario Geiger Leonardo Petrini M. Wyart DRL 31 11 0 30 Dec 2020
Analysis of the Scalability of a Deep-Learning Network for Steganography "Into the Wild" Hugo Ruiz Marc Chaumont Mehdi Yedroudj A. Amara Frédéric Comby Gérard Subsol 29 9 0 29 Dec 2020
*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task Dmitry Tsarkov Tibor Tihon Nathan Scales Nikola Momchev Danila Sinopalnikov Nathanael Scharli 18 17 0 15 Dec 2020
Generalization bounds for deep learning Guillermo Valle Pérez A. Louis BDL 13 44 0 07 Dec 2020
Learning Curves for Drug Response Prediction in Cancer Cell Lines A. Partin Thomas Brettin Yvonne A. Evrard Yitan Zhu H. Yoo ... Austin R. Clyde Maulik Shukla Michael Fonstein J. Doroshow Rick L. Stevens 18 19 0 25 Nov 2020
Video Big Data Analytics in the Cloud: A Reference Architecture, Survey, Opportunities, and Open Research Issues A. Alam I. Ullah Young-Koo Lee 42 22 0 16 Nov 2020
Video Big Data Analytics in the Cloud: Research Issues and Challenges A. Alam S. Khalid Muhammad Numan Khan Tariq Habib Afridi I. Ullah Young-Koo Lee 18 1 0 05 Nov 2020
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference Michael Lui Yavuz Yetim Özgür Özkan Zhuoran Zhao Shin-Yeh Tsai Carole-Jean Wu Mark Hempstead GNN BDL LRM 22 51 0 04 Nov 2020
CopyPaste: An Augmentation Method for Speech Emotion Recognition R. Pappagari Jesús Villalba Piotr Żelasko Laureano Moro Velázquez Najim Dehak 19 39 0 27 Oct 2020
Are wider nets better given the same number of parameters? A. Golubeva Behnam Neyshabur Guy Gur-Ari 27 44 0 27 Oct 2020
The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research N. Ahmed Muntasir Wahed 25 106 0 22 Oct 2020
Deep Learning is Singular, and That's Good Daniel Murfet Susan Wei Biwei Huang Hui Li Jesse Gell-Redman T. Quella UQCV 24 26 0 22 Oct 2020
Transferable Graph Optimizers for ML Compilers Yanqi Zhou Sudip Roy AmirAli Abdolrashidi Daniel Wong Peter C. Ma ... Mangpo Phitchaya Phothilimtha Shen Wang Anna Goldie Azalia Mirhoseini James Laudon GNN 8 53 0 21 Oct 2020
Learning Curves for Analysis of Deep Networks Derek Hoiem Tanmay Gupta Zhizhong Li Michal Shlapentokh-Rothman 18 24 0 21 Oct 2020
Small Data, Big Decisions: Model Selection in the Small-Data Regime J. Bornschein Francesco Visin Simon Osindero 21 36 0 26 Sep 2020
Pruning Convolutional Filters using Batch Bridgeout Najeeb Khan Ian Stavness 28 3 0 23 Sep 2020
Action-Based Representation Learning for Autonomous Driving Yi Xiao Felipe Codevilla C. Pal Antonio M. López 22 10 0 21 Aug 2020
Geometric compression of invariant manifolds in neural nets J. Paccolat Leonardo Petrini Mario Geiger Kevin Tyloo M. Wyart MLT 55 34 0 22 Jul 2020
Add a SideNet to your MainNet Adrien Morisot 14 0 0 14 Jul 2020
The Computational Limits of Deep Learning Neil C. Thompson Kristjan Greenewald Keeheon Lee Gabriel F. Manso VLM 26 508 0 10 Jul 2020
Is SGD a Bayesian sampler? Well, almost Chris Mingard Guillermo Valle Pérez Joar Skalse A. Louis BDL 23 51 0 26 Jun 2020
On the Predictability of Pruning Across Scales Jonathan S. Rosenfeld Jonathan Frankle Michael Carbin Nir Shavit 25 37 0 18 Jun 2020
To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks Sinong Wang Madian Khabsa Hao Ma 18 26 0 15 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 77 40,200 0 28 May 2020
How fine can fine-tuning be? Learning efficient language models Evani Radiya-Dixit Xin Wang 16 63 0 24 Apr 2020
Embedded Large-Scale Handwritten Chinese Character Recognition Youssouf Chherawala Hans J. G. A. Dolfing Ryan S. Dixon J. Bellegarda 11 5 0 13 Apr 2020
Leveraging GANs to Improve Continuous Path Keyboard Input Models Akash Mehra J. Bellegarda Ojas Bapat Partha Lal Xin Wang 8 8 0 06 Apr 2020
SuperMix: Supervising the Mixing Data Augmentation Ali Dabouei Sobhan Soleymani Fariborz Taherkhani Nasser M. Nasrabadi 19 98 0 10 Mar 2020
Slice Tuner: A Selective Data Acquisition Framework for Accurate and Fair Machine Learning Models Ki Hyun Tae Steven Euijong Whang 23 39 0 10 Mar 2020
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar Cengiz Pehlevan 146 201 0 07 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,505 0 23 Jan 2020
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence Kihyuk Sohn David Berthelot Chun-Liang Li Zizhao Zhang Nicholas Carlini E. D. Cubuk Alexey Kurakin Han Zhang Colin Raffel AAML 104 3,467 0 21 Jan 2020
Designing for the Long Tail of Machine Learning Martin Lindvall J. Molin HAI 6 2 0 21 Jan 2020
Social and Governance Implications of Improved Data Efficiency Aaron David Tucker Markus Anderljung Allan Dafoe 16 14 0 14 Jan 2020
Value-laden Disciplinary Shifts in Machine Learning Ravit Dotan S. Milli AILaw 27 48 0 03 Dec 2019
Using Error Decay Prediction to Overcome Practical Issues of Deep Active Learning for Named Entity Recognition Haw-Shiuan Chang Shankar Vembu Sunil Mohan Rheeya Uppaal Andrew McCallum 4 3 0 17 Nov 2019
An empirical study of the relation between network architecture and complexity Emir Konuk Kevin Smith 21 7 0 11 Nov 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 126 19,493 0 23 Oct 2019
Improving Differentially Private Models with Active Learning Zhengli Zhao Nicolas Papernot Sameer Singh N. Polyzotis Augustus Odena SyDa 6 5 0 02 Oct 2019
GDP: Generalized Device Placement for Dataflow Graphs Yanqi Zhou Sudip Roy AmirAli Abdolrashidi Daniel Wong Peter C. Ma ... Ming Zhong Hanxiao Liu Anna Goldie Azalia Mirhoseini James Laudon GNN 27 38 0 28 Sep 2019
A Constructive Prediction of the Generalization Error Across Scales Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit 36 205 0 27 Sep 2019