Gradient Descent Maximizes the Margin of Homogeneous Neural Networks

13 June 2019

Papers citing "Gradient Descent Maximizes the Margin of Homogeneous Neural Networks"

50 / 245 papers shown

Title
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability Mathieu Even Scott Pesme Suriya Gunasekar Nicolas Flammarion 28 16 0 17 Feb 2023
The Missing Margin: How Sample Corruption Affects Distance to the Boundary in ANNs Marthinus W. Theunissen Coenraad Mouton Marelie Hattingh Davel 13 1 0 14 Feb 2023
Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillation Noel Loo Ramin Hasani Mathias Lechner Alexander Amini Daniela Rus DD 42 5 0 02 Feb 2023
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing Jikai Jin Zhiyuan Li Kaifeng Lyu S. Du Jason D. Lee MLT 54 34 0 27 Jan 2023
Understanding Difficulty-based Sample Weighting with a Universal Difficulty Measure Xiaoling Zhou Ou Wu Weiyao Zhu Ziyang Liang 27 2 0 12 Jan 2023
Iterative regularization in classification via hinge loss diagonal descent Vassilis Apidopoulos T. Poggio Lorenzo Rosasco S. Villa 24 2 0 24 Dec 2022
Improved Convergence Guarantees for Shallow Neural Networks A. Razborov ODL 27 1 0 05 Dec 2022
Neural networks trained with SGD learn distributions of increasing complexity Maria Refinetti Alessandro Ingrosso Sebastian Goldt UQCV 32 41 0 21 Nov 2022
Mechanistic Mode Connectivity Ekdeep Singh Lubana Eric J. Bigelow Robert P. Dick David M. Krueger Hidenori Tanaka 32 45 0 15 Nov 2022
Regression as Classification: Influence of Task Formulation on Neural Network Features Lawrence Stewart Francis R. Bach Quentin Berthet Jean-Philippe Vert 29 24 0 10 Nov 2022
Do highly over-parameterized neural networks generalize since bad solutions are rare? Julius Martinetz T. Martinetz 27 1 0 07 Nov 2022
Learning Single-Index Models with Shallow Neural Networks A. Bietti Joan Bruna Clayton Sanford M. Song 167 67 0 27 Oct 2022
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 40 49 0 25 Oct 2022
Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets Edo Cohen-Karlik Itamar Menuhin-Gruman Raja Giryes Nadav Cohen Amir Globerson 25 4 0 25 Oct 2022
Vision Transformers provably learn spatial structure Samy Jelassi Michael E. Sander Yuan-Fang Li ViT MLT 34 74 0 13 Oct 2022
Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data Spencer Frei Gal Vardi Peter L. Bartlett Nathan Srebro Wei Hu MLT 30 38 0 13 Oct 2022
Optimal AdaBoost Converges Conor Snedeker 11 0 0 11 Oct 2022
C-Mixup: Improving Generalization in Regression Huaxiu Yao Yiping Wang Linjun Zhang James Zou Chelsea Finn UQCV OOD 35 55 0 11 Oct 2022
SGD with Large Step Sizes Learns Sparse Features Maksym Andriushchenko Aditya Varre Loucas Pillaud-Vivien Nicolas Flammarion 45 56 0 11 Oct 2022
Boosting Adversarial Robustness From The Perspective of Effective Margin Regularization Ziquan Liu Antoni B. Chan AAML 27 5 0 11 Oct 2022
The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks D. Kunin Atsushi Yamamura Chao Ma Surya Ganguli 17 20 0 07 Oct 2022
Testing predictions of representation cost theory with CNNs Charles Godfrey Elise Bishoff Myles Mckay Davis Brown Grayson Jorgenson Henry Kvinge E. Byler 24 0 0 03 Oct 2022
Magnitude and Angle Dynamics in Training Single ReLU Neurons Sangmin Lee Byeongsu Sim Jong Chul Ye MLT 96 6 0 27 Sep 2022
Importance Tempering: Group Robustness for Overparameterized Models Yiping Lu Wenlong Ji Zachary Izzo Lexing Ying 42 7 0 19 Sep 2022
On the Implicit Bias in Deep-Learning Algorithms Gal Vardi FedML AI4CE 34 72 0 26 Aug 2022
Imbalance Trouble: Revisiting Neural-Collapse Geometry Christos Thrampoulidis Ganesh Ramachandra Kini V. Vakilian Tina Behnia 24 68 0 10 Aug 2022
Feature selection with gradient descent on two-layer networks in low-rotation regimes Matus Telgarsky MLT 28 16 0 04 Aug 2022
Adaptive Gradient Methods at the Edge of Stability Jeremy M. Cohen Behrooz Ghorbani Shankar Krishnan Naman Agarwal Sourabh Medapati ... Daniel Suo David E. Cardoze Zachary Nado George E. Dahl Justin Gilmer ODL 29 50 0 29 Jul 2022
Towards understanding how momentum improves generalization in deep learning Samy Jelassi Yuanzhi Li ODL MLT AI4CE 27 31 0 13 Jul 2022
Synergy and Symmetry in Deep Learning: Interactions between the Data, Model, and Inference Algorithm Lechao Xiao Jeffrey Pennington 34 10 0 11 Jul 2022
Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent Zhiyuan Li Tianhao Wang Jason D. Lee Sanjeev Arora 37 27 0 08 Jul 2022
Label noise (stochastic) gradient descent implicitly solves the Lasso for quadratic parametrisation Loucas Pillaud-Vivien J. Reygner Nicolas Flammarion NoLa 33 31 0 20 Jun 2022
Max-Margin Works while Large Margin Fails: Generalization without Uniform Convergence Margalit Glasgow Colin Wei Mary Wootters Tengyu Ma 38 5 0 16 Jun 2022
Reconstructing Training Data from Trained Neural Networks Niv Haim Gal Vardi Gilad Yehudai Ohad Shamir Michal Irani 40 132 0 15 Jun 2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 40 69 0 14 Jun 2022
The Slingshot Mechanism: An Empirical Study of Adaptive Optimizers and the Grokking Phenomenon Vimal Thilak Etai Littwin Shuangfei Zhai Omid Saremi Roni Paiss J. Susskind 26 61 0 10 Jun 2022
Explicit Regularization in Overparametrized Models via Noise Injection Antonio Orvieto Anant Raj Hans Kersting Francis R. Bach 10 26 0 09 Jun 2022
Neural Collapse: A Review on Modelling Principles and Generalization Vignesh Kothapalli 23 71 0 08 Jun 2022
Adversarial Reprogramming Revisited Matthias Englert R. Lazic AAML 26 8 0 07 Jun 2022
Understanding Deep Learning via Decision Boundary Shiye Lei Fengxiang He Yancheng Yuan Dacheng Tao 19 13 0 03 Jun 2022
Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs Etienne Boursier Loucas Pillaud-Vivien Nicolas Flammarion ODL 21 58 0 02 Jun 2022
Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measures Emmanuel Abbe Samy Bengio Elisabetta Cornacchia Jon M. Kleinberg Aryo Lotfi M. Raghu Chiyuan Zhang MLT 16 10 0 26 May 2022
Mirror Descent Maximizes Generalized Margin and Can Be Implemented Efficiently Haoyuan Sun Kwangjun Ahn Christos Thrampoulidis Navid Azizan OOD 9 21 0 25 May 2022
On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias Itay Safran Gal Vardi Jason D. Lee MLT 59 23 0 18 May 2022
The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning Zixin Wen Yuanzhi Li SSL 27 34 0 12 May 2022
High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Models Tengyuan Liang Subhabrata Sen Pragya Sur 39 7 0 09 Apr 2022
Fast Rates for Noisy Interpolation Require Rethinking the Effects of Inductive Bias Konstantin Donhauser Nicolò Ruggeri Stefan Stojanovic Fanny Yang 18 21 0 07 Mar 2022
Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimization I Zaghloul Amir Roi Livni Nathan Srebro 30 6 0 27 Feb 2022
From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality Fusheng Liu Haizhao Yang Soufiane Hayou Qianxiao Li AI4CE 18 2 0 22 Feb 2022
Stochastic linear optimization never overfits with quadratically-bounded losses on general data Matus Telgarsky 11 11 0 14 Feb 2022