Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models

26 October 2020

Papers citing "Memorizing without overfitting: Bias, variance, and interpolation in over-parameterized models"

44 / 44 papers shown

Title
Breaking Neural Network Scaling Laws with Modularity Akhilan Boopathy Sunshine Jiang William Yue Jaedong Hwang Abhiram Iyer Ila Fiete OOD 104 2 0 09 Sep 2024
The Geometry of Over-parameterized Regression and Adversarial Perturbations J. Rocks Pankaj Mehta AAML 27 8 0 25 Mar 2021
The Perturbative Resolvent Method: spectral densities of random matrix ensembles via perturbation theory Wenping Cui J. Rocks Pankaj Mehta 20 5 0 01 Dec 2020
Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition Ben Adlam Jeffrey Pennington UD 67 93 0 04 Nov 2020
Wide flat minima and optimal generalization in classifying high-dimensional Gaussian mixtures Carlo Baldassi Enrico M. Malatesta Matteo Negri R. Zecchina 25 13 0 27 Oct 2020
What causes the test error? Going beyond bias-variance via ANOVA Licong Lin Yan Sun 44 34 0 11 Oct 2020
Benign Overfitting and Noisy Features Zhu Li Weijie Su Dino Sejdinovic 32 22 0 06 Aug 2020
Entropic gradient descent algorithms and wide flat minima Fabrizio Pittorino Carlo Lucibello Christoph Feinauer Gabriele Perugini Carlo Baldassi Elizaveta Demyanenko R. Zecchina ODL MLT 66 33 0 14 Jun 2020
A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent Zhenyu Liao Romain Couillet Michael W. Mahoney 69 89 0 09 Jun 2020
Triple descent and the two kinds of overfitting: Where & why do they appear? Stéphane dÁscoli Levent Sagun Giulio Biroli 39 80 0 05 Jun 2020
A Brief Prehistory of Double Descent Marco Loog T. Viering A. Mey Jesse H. Krijthe David Tax 43 69 0 07 Apr 2020
Double Trouble in Double Descent : Bias and Variance(s) in the Lazy Regime Stéphane dÁscoli Maria Refinetti Giulio Biroli Florent Krzakala 139 152 0 02 Mar 2020
Rethinking Bias-Variance Trade-off for Generalization of Neural Networks Zitong Yang Yaodong Yu Chong You Jacob Steinhardt Yi-An Ma 65 184 0 26 Feb 2020
Generalisation error in learning with random features and the hidden manifold model Federica Gerace Bruno Loureiro Florent Krzakala M. Mézard Lenka Zdeborová 62 169 0 21 Feb 2020
Implicit Regularization of Random Feature Models Arthur Jacot Berfin Simsek Francesco Spadaro Clément Hongler Franck Gabriel 57 83 0 19 Feb 2020
Analytic Study of Double Descent in Binary Classification: The Impact of Loss Ganesh Ramachandra Kini Christos Thrampoulidis 55 52 0 30 Jan 2020
More Data Can Hurt for Linear Regression: Sample-wise Double Descent Preetum Nakkiran 42 68 0 16 Dec 2019
Exact expressions for double descent and implicit regularization via surrogate random design Michal Derezinski Feynman T. Liang Michael W. Mahoney 55 78 0 10 Dec 2019
Deep Double Descent: Where Bigger Models and More Data Hurt Preetum Nakkiran Gal Kaplun Yamini Bansal Tristan Yang Boaz Barak Ilya Sutskever 119 935 0 04 Dec 2019
A Model of Double Descent for High-dimensional Binary Linear Classification Zeyu Deng A. Kammoun Christos Thrampoulidis 74 146 0 13 Nov 2019
Non-Gaussian processes and neural networks at finite widths Sho Yaida 76 87 0 30 Sep 2019
The generalization error of random features regression: Precise asymptotics and double descent curve Song Mei Andrea Montanari 80 634 0 14 Aug 2019
Benign Overfitting in Linear Regression Peter L. Bartlett Philip M. Long Gábor Lugosi Alexander Tsigler MLT 67 777 0 26 Jun 2019
Understanding overfitting peaks in generalization error: Analytical risk curves for $l_2$ and $l_1$ penalized interpolation P. Mitra 48 50 0 09 Jun 2019
A New Look at an Old Problem: A Universal Learning Approach to Linear Regression Koby Bibas Yaniv Fogel M. Feder 36 34 0 12 May 2019
Harmless interpolation of noisy data in regression Vidya Muthukumar Kailas Vodrahalli Vignesh Subramanian A. Sahai 65 204 0 21 Mar 2019
Surprises in High-Dimensional Ridgeless Least Squares Interpolation Trevor Hastie Andrea Montanari Saharon Rosset Robert Tibshirani 155 743 0 19 Mar 2019
Two models of double descent for weak features M. Belkin Daniel J. Hsu Ji Xu 85 376 0 18 Mar 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 178 1,097 0 18 Feb 2019
Scaling description of generalization with number of parameters in deep learning Mario Geiger Arthur Jacot S. Spigler Franck Gabriel Levent Sagun Stéphane dÁscoli Giulio Biroli Clément Hongler Matthieu Wyart 76 195 0 06 Jan 2019
Reconciling modern machine learning practice and the bias-variance trade-off M. Belkin Daniel J. Hsu Siyuan Ma Soumik Mandal 195 1,638 0 28 Dec 2018
A jamming transition from under- to over-parametrization affects loss landscape and generalization S. Spigler Mario Geiger Stéphane dÁscoli Levent Sagun Giulio Biroli Matthieu Wyart 58 153 0 22 Oct 2018
An analytic theory of generalization dynamics and transfer learning in deep linear networks Andrew Kyle Lampinen Surya Ganguli OOD 62 131 0 27 Sep 2018
The jamming transition as a paradigm to understand the loss landscape of deep neural networks Mario Geiger S. Spigler Stéphane dÁscoli Levent Sagun Marco Baity-Jesi Giulio Biroli Matthieu Wyart 51 143 0 25 Sep 2018
Just Interpolate: Kernel "Ridgeless" Regression Can Generalize Tengyuan Liang Alexander Rakhlin 58 353 0 01 Aug 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 234 3,191 0 20 Jun 2018
A high-bias, low-variance introduction to Machine Learning for physicists Pankaj Mehta Marin Bukov Ching-Hao Wang A. G. Day C. Richardson Charles K. Fisher D. Schwab AI4CE 79 873 0 23 Mar 2018
High-dimensional dynamics of generalization error in neural networks Madhu S. Advani Andrew M. Saxe AI4CE 128 469 0 10 Oct 2017
Optimal Errors and Phase Transitions in High-Dimensional Generalized Linear Models Jean Barbier Florent Krzakala N. Macris Léo Miolane Lenka Zdeborová 68 265 0 10 Aug 2017
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks Levent Sagun Utku Evci V. U. Güney Yann N. Dauphin Léon Bottou 54 418 0 14 Jun 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 310 4,623 0 10 Nov 2016
Entropy-SGD: Biasing Gradient Descent Into Wide Valleys Pratik Chaudhari A. Choromańska Stefano Soatto Yann LeCun Carlo Baldassi C. Borgs J. Chayes Levent Sagun R. Zecchina ODL 94 773 0 06 Nov 2016
An Analysis of Deep Neural Network Models for Practical Applications A. Canziani Adam Paszke Eugenio Culurciello 85 1,166 0 24 May 2016
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization Yann N. Dauphin Razvan Pascanu Çağlar Gülçehre Kyunghyun Cho Surya Ganguli Yoshua Bengio ODL 123 1,383 0 10 Jun 2014