Deep Learning Scaling is Predictable, Empirically

1 December 2017

Sharan Narang

Md. Mostofa Ali Patwary

Yang Yang

Yanqi Zhou

ArXiv PDF HTML

Papers citing "Deep Learning Scaling is Predictable, Empirically"

50 / 386 papers shown

Title
Human-Algorithm Collaboration: Achieving Complementarity and Avoiding Unfairness Kate Donahue Alexandra Chouldechova K. Kenthapadi FaML FedML 52 50 0 17 Feb 2022
Compute Trends Across Three Eras of Machine Learning J. Sevilla Lennart Heim A. Ho T. Besiroglu Marius Hobbhahn Pablo Villalobos 39 269 0 11 Feb 2022
Failure and success of the spectral bias prediction for Kernel Ridge Regression: the case of low-dimensional data Umberto M. Tomasini Antonio Sclocchi M. Wyart 15 12 0 07 Feb 2022
DASHA: Distributed Nonconvex Optimization with Communication Compression, Optimal Oracle Complexity, and No Client Synchronization A. Tyurin Peter Richtárik 41 18 0 02 Feb 2022
Reducing the Amount of Real World Data for Object Detector Training with Synthetic Data Sven Burdorf Karoline Plum Daniel Hasenklever 18 4 0 31 Jan 2022
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions Hugo Cui Bruno Loureiro Florent Krzakala Lenka Zdeborová 46 10 0 29 Jan 2022
A Transferable Approach for Partitioning Machine Learning Models on Multi-Chip-Modules Xinfeng Xie Prakash Prabhu Ulysse Beaugnon P. Phothilimthana Sudip Roy Azalia Mirhoseini E. Brevdo James Laudon Yanqi Zhou 30 5 0 07 Dec 2021
Models of fairness in federated learning Kate Donahue Jon M. Kleinberg FedML 60 9 0 01 Dec 2021
AugLiChem: Data Augmentation Library of Chemical Structures for Machine Learning Rishikesh Magar Yuyang Wang Cooper Lorsung Chen Liang Hariharan Ramasubramanian Peiyuan Li A. Farimani 28 27 0 30 Nov 2021
Turing-Universal Learners with Optimal Scaling Laws Preetum Nakkiran 29 2 0 09 Nov 2021
Learning curves for Gaussian process regression with power-law priors and targets Hui Jin P. Banerjee Guido Montúfar 14 17 0 23 Oct 2021
The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Fail Sam Bowman OffRL 24 45 0 15 Oct 2021
Training Deep Neural Networks with Joint Quantization and Pruning of Weights and Activations Xinyu Zhang Ian Colbert Ken Kreutz-Delgado Srinjoy Das MQ 32 11 0 15 Oct 2021
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers Gabriele Prato Simon Guiroy Ethan Caballero Irina Rish Sarath Chandar VLM 34 11 0 13 Oct 2021
RankingMatch: Delving into Semi-Supervised Learning with Consistency Regularization and Ranking Loss Trung Q. Tran Mingu Kang Daeyoung Kim 16 2 0 09 Oct 2021
Unsupervised Selective Labeling for More Effective Semi-Supervised Learning Xudong Wang Long Lian Stella X. Yu 194 33 0 06 Oct 2021
Max and Coincidence Neurons in Neural Networks Albert Lee Kang L. Wang 6 1 0 04 Oct 2021
Noise2Recon: Enabling Joint MRI Reconstruction and Denoising with Semi-Supervised and Self-Supervised Learning Arjun D Desai Batu Mehmet Ozturkler Christopher M. Sandino R. Boutin M. Willis S. Vasanawala B. Hargreaves Christopher Ré John M. Pauly Akshay S. Chaudhari 27 3 0 30 Sep 2021
Robust Temporal Ensembling for Learning with Noisy Labels Abel Brown Benedikt Schifferer R. DiPietro NoLa OOD 14 0 0 29 Sep 2021
Unsolved Problems in ML Safety Dan Hendrycks Nicholas Carlini John Schulman Jacob Steinhardt 186 276 0 28 Sep 2021
Is the Number of Trainable Parameters All That Actually Matters? A. Chatelain Amine Djeghri Daniel Hesslow Julien Launay Iacopo Poli 51 7 0 24 Sep 2021
Scaling Laws for Neural Machine Translation Behrooz Ghorbani Orhan Firat Markus Freitag Ankur Bapna M. Krikun Xavier Garcia Ciprian Chelba Colin Cherry 40 99 0 16 Sep 2021
Formalizing and Estimating Distribution Inference Risks Anshuman Suri David Evans MIACV 45 51 0 13 Sep 2021
Compute and Energy Consumption Trends in Deep Learning Inference Radosvet Desislavov Fernando Martínez-Plumed José Hernández-Orallo 35 113 0 12 Sep 2021
Why and How Governments Should Monitor AI Development Jess Whittlestone Jack Clark 35 30 0 28 Aug 2021
A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective? Hiroaki Mikami Kenji Fukumizu Shogo Murai Shuji Suzuki Yuta Kikuchi Taiji Suzuki S. Maeda Kohei Hayashi 40 12 0 25 Aug 2021
Scalable Bayesian transport maps for high-dimensional non-Gaussian spatial fields Matthias Katzfuss Florian Schafer OT 32 14 0 09 Aug 2021
On The State of Data In Computer Vision: Human Annotations Remain Indispensable for Developing Deep Learning Models Z. Emam Andrew Kondrich Sasha Harrison Felix Lau Yushi Wang Aerin Kim E. Branson VLM 38 11 0 31 Jul 2021
Dataset Distillation with Infinitely Wide Convolutional Networks Timothy Nguyen Roman Novak Lechao Xiao Jaehoon Lee DD 51 231 0 27 Jul 2021
Learning to Limit Data Collection via Scaling Laws: A Computational Interpretation for the Legal Principle of Data Minimization Divya Shanmugam Samira Shabanian Fernando Diaz Michèle Finck Joanna Biega 8 18 0 16 Jul 2021
A Dual-Purpose Deep Learning Model for Auscultated Lung and Tracheal Sound Analysis Based on Mixed Set Training Fu-Shun Hsu Shang-Ran Huang Chang-Fu Su Chien-Wen Huang Yuan-Ren Cheng ... Nian-Jhen Lin Wan-Ling Tsai Ching-Shiang Lu Chuan Chen F. Lai 29 5 0 09 Jul 2021
Structured Model Pruning of Convolutional Networks on Tensor Processing Units Kongtao Chen Ken Franko Ruoxin Sang CVBM 11 59 0 09 Jul 2021
Meta-learning Amidst Heterogeneity and Ambiguity Kyeongryeol Go Seyoung Yun 34 1 0 05 Jul 2021
Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records Yikuan Li M. Mamouei G. Salimi-Khorshidi Shishir Rao A. Hassaine D. Canoy Thomas Lukasiewicz K. Rahimi 26 76 0 21 Jun 2021
Data Optimisation for a Deep Learning Recommender System Gustav Hertz Sandhya Sachidanandan Balázs Tóth Emil S. Jørgensen Martin Tegnér 22 0 0 21 Jun 2021
Locality defeats the curse of dimensionality in convolutional teacher-student scenarios Alessandro Favero Francesco Cagnetta M. Wyart 30 31 0 16 Jun 2021
Towards Costless Model Selection in Contextual Bandits: A Bias-Variance Perspective Sanath Kumar Krishnamurthy Adrienne Margaret Propp Susan Athey 25 3 0 11 Jun 2021
Redundant representations help generalization in wide neural networks Diego Doimo Aldo Glielmo Sebastian Goldt Alessandro Laio AI4CE 33 9 0 07 Jun 2021
Self-Supervision is All You Need for Solving Rubik's Cube Kyo Takano 15 1 0 06 Jun 2021
Search Spaces for Neural Model Training Darko Stosic Dusan Stosic 28 4 0 27 May 2021
A Theoretical-Empirical Approach to Estimating Sample Complexity of DNNs Devansh Bisla Apoorva Nandini Saridena A. Choromańska 30 8 0 05 May 2021
Accelerating Sparse Deep Neural Networks Asit K. Mishra J. Latorre Jeff Pool Darko Stosic Dusan Stosic Ganesh Venkatesh Chong Yu Paulius Micikevicius 22 221 0 16 Apr 2021
Scaling Scaling Laws with Board Games Andrew Jones 8 39 0 07 Apr 2021
Vulnerability Due to Training Order in Split Learning Harshit Madaan M. Gawali V. Kulkarni Aniruddha Pant FedML 24 6 0 26 Mar 2021
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark Nicholas Lourie Ronan Le Bras Chandra Bhagavatula Yejin Choi LRM 30 137 0 24 Mar 2021
The Shape of Learning Curves: a Review T. Viering Marco Loog 18 122 0 19 Mar 2021
The Low-Rank Simplicity Bias in Deep Networks Minyoung Huh H. Mobahi Richard Y. Zhang Brian Cheung Pulkit Agrawal Phillip Isola 27 109 0 18 Mar 2021
Is it enough to optimize CNN architectures on ImageNet? Lukas Tuggener Jürgen Schmidhuber Thilo Stadelmann 33 23 0 16 Mar 2021
Revisiting ResNets: Improved Training and Scaling Strategies Irwan Bello W. Fedus Xianzhi Du E. D. Cubuk A. Srinivas Nayeon Lee Jonathon Shlens Barret Zoph 29 298 0 13 Mar 2021
Integration of Convolutional Neural Networks in Mobile Applications Roger Creus Castanyer Silverio Martínez-Fernández Xavier Franch 29 12 0 11 Mar 2021