Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning

2 October 2018

Papers citing "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning"

27 / 127 papers shown

Title
Halting Time is Predictable for Large Models: A Universality Property and Average-case Analysis Courtney Paquette B. V. Merrienboer Elliot Paquette Fabian Pedregosa 29 25 0 08 Jun 2020
Achieving Online Regression Performance of LSTMs with Simple RNNs Nuri Mert Vural Fatih Ilhan Selim F. Yilmaz Salih Ergüt Suleyman Serdar Kozat 11 13 0 16 May 2020
Optimization in Machine Learning: A Distribution Space Approach Yongqiang Cai Qianxiao Li Zuowei Shen 25 1 0 18 Apr 2020
The Implicit Regularization of Stochastic Gradient Flow for Least Squares Alnur Ali Yan Sun R. Tibshirani 22 76 0 17 Mar 2020
On Alignment in Deep Linear Neural Networks Adityanarayanan Radhakrishnan Eshaan Nichani D. Bernstein Caroline Uhler 17 2 0 13 Mar 2020
Predicting trends in the quality of state-of-the-art neural networks without access to training or testing data Charles H. Martin Tongsu Peng Peng Michael W. Mahoney 36 101 0 17 Feb 2020
Average-case Acceleration Through Spectral Density Estimation Fabian Pedregosa Damien Scieur 14 12 0 12 Feb 2020
Regularized Evolutionary Population-Based Training J. Liang Santiago Gonzalez H. Shahrzad Risto Miikkulainen 22 9 0 11 Feb 2020
Self-Orthogonality Module: A Network Architecture Plug-in for Learning Orthogonal Filters Ziming Zhang Wenchi Ma Yuanwei Wu Guanghui Wang 40 10 0 05 Jan 2020
Exact expressions for double descent and implicit regularization via surrogate random design Michal Derezinski Feynman T. Liang Michael W. Mahoney 27 77 0 10 Dec 2019
Implicit Regularization and Convergence for Weight Normalization Xiaoxia Wu Yan Sun Tongzheng Ren Shanshan Wu Zhiyuan Li Suriya Gunasekar Rachel A. Ward Qiang Liu 20 21 0 18 Nov 2019
Periodic Spectral Ergodicity: A Complexity Measure for Deep Neural Networks and Neural Architecture Search Mehmet Süzen J. Cerdà C. Weber 13 1 0 10 Nov 2019
Detecting Underspecification with Local Ensembles David Madras James Atwood Alexander DÁmour 33 4 0 21 Oct 2019
Compression based bound for non-compressed network: unified generalization error analysis of large compressible deep neural network Taiji Suzuki Hiroshi Abe Tomoaki Nishimura AI4CE 25 43 0 25 Sep 2019
PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization Thijs Vogels Sai Praneeth Karimireddy Martin Jaggi 19 317 0 31 May 2019
On Dropout and Nuclear Norm Regularization Poorya Mianjy R. Arora 24 23 0 28 May 2019
HARK Side of Deep Learning -- From Grad Student Descent to Automated Machine Learning O. Gencoglu M. Gils E. Guldogan Chamin Morikawa Mehmet Süzen M. Gruber J. Leinonen H. Huttunen 11 36 0 16 Apr 2019
Deep Fundamental Factor Models M. Dixon Nicholas G. Polson 26 9 0 18 Mar 2019
Inefficiency of K-FAC for Large Batch Size Training Linjian Ma Gabe Montague Jiayu Ye Z. Yao A. Gholami Kurt Keutzer Michael W. Mahoney 24 24 0 14 Mar 2019
Combining learning rate decay and weight decay with complexity gradient descent - Part I Pierre Harvey Richemond Yike Guo 25 4 0 07 Feb 2019
Cross-Entropy Loss and Low-Rank Features Have Responsibility for Adversarial Examples Kamil Nar Orhan Ocal S. Shankar Sastry Kannan Ramchandran AAML 27 54 0 24 Jan 2019
Heavy-Tailed Universality Predicts Trends in Test Accuracies for Very Large Pre-Trained Deep Neural Networks Charles H. Martin Michael W. Mahoney 15 55 0 24 Jan 2019
Traditional and Heavy-Tailed Self Regularization in Neural Network Models Charles H. Martin Michael W. Mahoney 21 120 0 24 Jan 2019
Cleaning large correlation matrices: tools from random matrix theory J. Bun J. Bouchaud M. Potters 32 262 0 25 Oct 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 308 2,892 0 15 Sep 2016
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 127 577 0 27 Feb 2015
The Loss Surfaces of Multilayer Networks A. Choromańska Mikael Henaff Michaël Mathieu Gerard Ben Arous Yann LeCun ODL 183 1,185 0 30 Nov 2014