Deep learning generalizes because the parameter-function map is biased towards simple functions

22 May 2018

Guillermo Valle Pérez

Papers citing "Deep learning generalizes because the parameter-function map is biased towards simple functions"

50 / 54 papers shown

Title
SPEX: Scaling Feature Interaction Explanations for LLMs Justin Singh Kang Landon Butler Abhineet Agarwal Yigit Efe Erginbas Ramtin Pedarsani Kannan Ramchandran Bin Yu VLM LRM 116 2 0 20 Feb 2025
A distributional simplicity bias in the learning dynamics of transformers Riccardo Rende Federica Gerace Alessandro Laio Sebastian Goldt 95 8 0 17 Feb 2025
Geometric Inductive Biases of Deep Networks: The Role of Data and Architecture Sajad Movahedi Antonio Orvieto Seyed-Mohsen Moosavi-Dezfooli AI4CE AAML 425 0 0 15 Oct 2024
SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning Hojoon Lee Dongyoon Hwang Donghu Kim Hyunseung Kim Jun Jet Tai K. Subramanian Peter R. Wurman Jaegul Choo Peter Stone Takuma Seno OffRL 109 14 0 13 Oct 2024
Bias of Stochastic Gradient Descent or the Architecture: Disentangling the Effects of Overparameterization of Neural Networks Amit Peleg Matthias Hein 45 0 0 04 Jul 2024
Neural Redshift: Random Networks are not Random Functions Damien Teney A. Nicolicioiu Valentin Hartmann Ehsan Abbasnejad 117 22 0 04 Mar 2024
Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes Greg Yang 80 197 0 28 Oct 2019
Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes Roman Novak Lechao Xiao Jaehoon Lee Yasaman Bahri Greg Yang Jiri Hron Daniel A. Abolafia Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 49 308 0 11 Oct 2018
Deep Convolutional Networks as shallow Gaussian Processes Adrià Garriga-Alonso C. Rasmussen Laurence Aitchison BDL UQCV 76 269 0 16 Aug 2018
Gaussian Process Behaviour in Wide Deep Neural Networks A. G. Matthews Mark Rowland Jiri Hron Richard Turner Zoubin Ghahramani BDL 124 553 0 30 Apr 2018
Simple random search provides a competitive approach to reinforcement learning Horia Mania Aurelia Guy Benjamin Recht 46 315 0 19 Mar 2018
On the importance of single directions for generalization Ari S. Morcos David Barrett Neil C. Rabinowitz M. Botvinick 51 331 0 19 Mar 2018
Energy-entropy competition and the effectiveness of stochastic gradient descent in machine learning Yao Zhang Andrew M. Saxe Madhu S. Advani A. Lee 47 60 0 05 Mar 2018
Essentially No Barriers in Neural Network Energy Landscape Felix Dräxler K. Veschgini M. Salmhofer Fred Hamprecht MoMe 97 430 0 02 Mar 2018
Stronger generalization bounds for deep nets via a compression approach Sanjeev Arora Rong Ge Behnam Neyshabur Yi Zhang MLT AI4CE 74 637 0 14 Feb 2018
To understand deep learning we need to understand kernel learning M. Belkin Siyuan Ma Soumik Mandal 44 418 0 05 Feb 2018
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors Gintare Karolina Dziugaite Daniel M. Roy MLT 52 145 0 26 Dec 2017
Deep Neuroevolution: Genetic Algorithms Are a Competitive Alternative for Training Deep Neural Networks for Reinforcement Learning F. Such Vashisht Madhavan Edoardo Conti Joel Lehman Kenneth O. Stanley Jeff Clune 78 690 0 18 Dec 2017
Size-Independent Sample Complexity of Neural Networks Noah Golowich Alexander Rakhlin Ohad Shamir 102 547 0 18 Dec 2017
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 111 1,086 0 01 Nov 2017
The Implicit Bias of Gradient Descent on Separable Data Daniel Soudry Elad Hoffer Mor Shpigel Nacson Suriya Gunasekar Nathan Srebro 83 908 0 27 Oct 2017
A Correspondence Between Random Neural Networks and Statistical Field Theory S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein 43 20 0 18 Oct 2017
A Bayesian Perspective on Generalization and Stochastic Gradient Descent Samuel L. Smith Quoc V. Le BDL 59 250 0 17 Oct 2017
Generalization in Deep Learning Kenji Kawaguchi L. Kaelbling Yoshua Bengio ODL 77 459 0 16 Oct 2017
High-dimensional dynamics of generalization error in neural networks Madhu S. Advani Andrew M. Saxe AI4CE 126 468 0 10 Oct 2017
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms Han Xiao Kashif Rasul Roland Vollgraf 178 8,807 0 25 Aug 2017
A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networks Behnam Neyshabur Srinadh Bhojanapalli Nathan Srebro 66 604 0 29 Jul 2017
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes Lei Wu Zhanxing Zhu E. Weinan ODL 50 220 0 30 Jun 2017
Exploring Generalization in Deep Learning Behnam Neyshabur Srinadh Bhojanapalli David A. McAllester Nathan Srebro FAtt 134 1,245 0 27 Jun 2017
Spectrally-normalized margin bounds for neural networks Peter L. Bartlett Dylan J. Foster Matus Telgarsky ODL 135 1,208 0 26 Jun 2017
A Closer Look at Memorization in Deep Networks Devansh Arpit Stanislaw Jastrzebski Nicolas Ballas David M. Krueger Emmanuel Bengio ... Tegan Maharaj Asja Fischer Aaron Courville Yoshua Bengio Simon Lacoste-Julien TDI 107 1,801 0 16 Jun 2017
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks Levent Sagun Utku Evci V. U. Güney Yann N. Dauphin Léon Bottou 41 416 0 14 Jun 2017
Stochastic Gradient Descent as Approximate Bayesian Inference Stephan Mandt Matthew D. Hoffman David M. Blei BDL 47 594 0 13 Apr 2017
Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data Gintare Karolina Dziugaite Daniel M. Roy 85 808 0 31 Mar 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 103 766 0 15 Mar 2017
Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks Peter L. Bartlett Nick Harvey Christopher Liaw Abbas Mehrabian 147 427 0 08 Mar 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 271 4,620 0 10 Nov 2016
Deep Information Propagation S. Schoenholz Justin Gilmer Surya Ganguli Jascha Narain Sohl-Dickstein 64 363 0 04 Nov 2016
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review T. Poggio H. Mhaskar Lorenzo Rosasco Brando Miranda Q. Liao 74 575 0 02 Nov 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 367 2,922 0 15 Sep 2016
Why does deep and cheap learning work so well? Henry W. Lin Max Tegmark David Rolnick 60 607 0 29 Aug 2016
Exponential expressivity in deep neural networks through transient chaos Ben Poole Subhaneil Lahiri M. Raghu Jascha Narain Sohl-Dickstein Surya Ganguli 83 587 0 16 Jun 2016
PAC-Bayesian Theory Meets Bayesian Inference Pascal Germain Francis R. Bach Alexandre Lacoste Simon Lacoste-Julien 54 182 0 27 May 2016
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford Luke Metz Soumith Chintala GAN OOD 232 13,968 0 19 Nov 2015
Train faster, generalize better: Stability of stochastic gradient descent Moritz Hardt Benjamin Recht Y. Singer 96 1,234 0 03 Sep 2015
Deep Neural Networks with Random Gaussian Weights: A Universal Classification Strategy? Raja Giryes Guillermo Sapiro A. Bronstein 83 187 0 30 Apr 2015
Norm-Based Capacity Control in Neural Networks Behnam Neyshabur Ryota Tomioka Nathan Srebro 236 583 0 27 Feb 2015
Explaining and Harnessing Adversarial Examples Ian Goodfellow Jonathon Shlens Christian Szegedy AAML GAN 182 18,922 0 20 Dec 2014
Deep Learning in Neural Networks: An Overview Jürgen Schmidhuber HAI 179 16,311 0 30 Apr 2014
On the Number of Linear Regions of Deep Neural Networks Guido Montúfar Razvan Pascanu Kyunghyun Cho Yoshua Bengio 77 1,250 0 08 Feb 2014