v1v2v3 (latest)

Width Provably Matters in Optimization for Deep Linear Neural Networks

24 January 2019

Papers citing "Width Provably Matters in Optimization for Deep Linear Neural Networks"

50 / 68 papers shown

Title
A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models Ziqing Xu Hancheng Min Salma Tarmoun Enrique Mallada Rene Vidal 123 0 0 16 May 2025
A new Input Convex Neural Network with application to options pricing Vincent Lemaire Gilles Pagès Christian Yeo 99 0 0 19 Nov 2024
Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks Zhenghao Xu Yuqing Wang T. Zhao Rachel Ward Molei Tao 63 1 0 12 Oct 2024
Swing-by Dynamics in Concept Learning and Compositional Generalization Yongyi Yang Core Francisco Park Ekdeep Singh Lubana Maya Okawa Wei Hu Hidenori Tanaka CoGe DiffM 57 1 0 10 Oct 2024
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 127 6 0 22 Sep 2024
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning Nadav Cohen Noam Razin 115 0 0 25 Aug 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 71 15 0 06 Jun 2024
Physics-Informed Neural Networks: Minimizing Residual Loss with Wide Networks and Effective Activations Nima Hosseini Dashtbayaz G. Farhani Boyu Wang Charles Ling 87 2 0 02 May 2024
NTK-Guided Few-Shot Class Incremental Learning Jingren Liu Zhong Ji Yanwei Pang YunLong Yu CLL 90 4 0 19 Mar 2024
Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensing Hong T.M. Chu Subhro Ghosh Chi Thanh Lam Soumendu Sundar Mukherjee 52 0 0 27 Feb 2024
Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation Markus Gross A. Raulf Christoph Räth 106 0 0 23 Nov 2023
On the Convergence of Federated Averaging under Partial Participation for Over-parameterized Neural Networks Xin Liu Wei Tao Dazhi Zhan Yu Pan Xin Ma Yu Ding Zhisong Pan FedML 70 0 0 09 Oct 2023
Towards Training Without Depth Limits: Batch Normalization Without Gradient Explosion Alexandru Meterez Amir Joudaki Francesco Orabona Alexander Immer Gunnar Rätsch Hadi Daneshmand 71 8 0 03 Oct 2023
Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space Zhengdao Chen 75 1 0 03 Jul 2023
Enhance Diffusion to Improve Robust Generalization Jianhui Sun Sanchit Sinha Aidong Zhang 69 4 0 05 Jun 2023
On a continuous time model of gradient descent dynamics and instability in deep learning Mihaela Rosca Yan Wu Chongli Qin Benoit Dherin 75 10 0 03 Feb 2023
Bayesian Interpolation with Deep Linear Networks Boris Hanin Alexander Zlokapa 137 26 0 29 Dec 2022
A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization JIAN-PENG Cao Chao Qian Yihui Huang Dicheng Chen Yuncheng Gao Jiyang Dong D. Guo X. Qu 114 1 0 29 Dec 2022
A Non-Asymptotic Analysis of Oversmoothing in Graph Neural Networks Xinyi Wu Zhengdao Chen W. Wang Ali Jadbabaie 111 45 0 21 Dec 2022
Infinite-width limit of deep linear neural networks Lénaïc Chizat Maria Colombo Xavier Fernández-Real Alessio Figalli 80 16 0 29 Nov 2022
Linear RNNs Provably Learn Linear Dynamic Systems Lifu Wang Tianyu Wang Shengwei Yi Bo Shen Bo Hu Xing Cao 39 0 0 19 Nov 2022
Deep Linear Networks can Benignly Overfit when Shallow Ones Do Niladri S. Chatterji Philip M. Long 96 8 0 19 Sep 2022
The Neural Race Reduction: Dynamics of Abstraction in Gated Networks Andrew M. Saxe Shagun Sodhani Sam Lewallen AI4CE 93 37 0 21 Jul 2022
Blessing of Nonconvexity in Deep Linear Models: Depth Flattens the Optimization Landscape Around the True Solution Jianhao Ma Salar Fattahi 75 5 0 15 Jul 2022
A note on Linear Bottleneck networks and their Transition to Multilinearity Libin Zhu Parthe Pandit M. Belkin MLT 79 0 0 30 Jun 2022
On Feature Learning in Neural Networks with Global Convergence Guarantees Zhengdao Chen Eric Vanden-Eijnden Joan Bruna MLT 84 13 0 22 Apr 2022
A Convergence Analysis of Nesterov's Accelerated Gradient Method in Training Deep Linear Neural Networks Xin Liu Wei Tao Zhisong Pan 23 9 0 18 Apr 2022
Convergence and Implicit Regularization Properties of Gradient Descent for Deep Residual Networks R. Cont Alain Rossier Renyuan Xu MLT 89 6 0 14 Apr 2022
Explicitising The Implicit Intrepretability of Deep Neural Networks Via Duality Chandrashekar Lakshminarayanan Ashutosh Kumar Singh A. Rajkumar AI4CE 64 1 0 01 Mar 2022
Global Convergence Analysis of Deep Linear Networks with A One-neuron Layer Kun Chen Dachao Lin Zhihua Zhang 56 1 0 08 Jan 2022
Imitating Deep Learning Dynamics via Locally Elastic Stochastic Differential Equations Jiayao Zhang Hua Wang Weijie J. Su 89 8 0 11 Oct 2021
Towards Demystifying Representation Learning with Non-contrastive Self-supervision Xiang Wang Xinlei Chen S. Du Yuandong Tian SSL 78 26 0 11 Oct 2021
Score-based Generative Neural Networks for Large-Scale Optimal Transport Max Daniels Tyler Maunu Paul Hand OT DiffM 75 72 0 07 Oct 2021
Convergence of gradient descent for learning linear neural networks Gabin Maxime Nguegnang Holger Rauhut Ulrich Terstiege MLT 54 18 0 04 Aug 2021
The loss landscape of deep linear neural networks: a second-order analysis El Mehdi Achour Franccois Malgouyres Sébastien Gerchinovitz ODL 65 10 0 28 Jul 2021
Continuous vs. Discrete Optimization of Deep Neural Networks Omer Elkabetz Nadav Cohen 111 44 0 14 Jul 2021
Ghosts in Neural Networks: Existence, Structure and Role of Infinite-Dimensional Null Space Sho Sonoda Isao Ishikawa Masahiro Ikeda BDL 51 9 0 09 Jun 2021
Muddling Label Regularization: Deep Learning for Tabular Datasets Karim Lounici Katia Méziani Benjamin Riu 76 6 0 08 Jun 2021
Convergence and Implicit Bias of Gradient Flow on Overparametrized Linear Networks Hancheng Min Salma Tarmoun René Vidal Enrique Mallada MLT 82 5 0 13 May 2021
Principal Components Bias in Over-parameterized Linear Models, and its Manifestation in Deep Neural Networks Guy Hacohen D. Weinshall 125 10 0 12 May 2021
Optimization of Graph Neural Networks: Implicit Acceleration by Skip Connections and More Depth Keyulu Xu Mozhi Zhang Stefanie Jegelka Kenji Kawaguchi GNN 53 78 0 10 May 2021
Initialization and Regularization of Factorized Neural Layers M. Khodak Neil A. Tenenholtz Lester W. Mackey Nicolò Fusi 147 57 0 03 May 2021
Learning with Neural Tangent Kernels in Near Input Sparsity Time A. Zandieh 47 0 0 01 Apr 2021
A Mathematical Principle of Deep Learning: Learn the Geodesic Curve in the Wasserstein Space Kuo Gai Shihua Zhang 96 8 0 18 Feb 2021
On the Theory of Implicit Deep Learning: Global Convergence with Implicit Layers Kenji Kawaguchi PINN 67 42 0 15 Feb 2021
Understanding self-supervised Learning Dynamics without Contrastive Pairs Yuandong Tian Xinlei Chen Surya Ganguli SSL 233 286 0 12 Feb 2021
The Connection Between Approximation, Depth Separation and Learnability in Neural Networks Eran Malach Gilad Yehudai Shai Shalev-Shwartz Ohad Shamir 85 20 0 31 Jan 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 167 80 0 08 Dec 2020
Which Minimizer Does My Neural Network Converge To? Manuel Nonnenmacher David Reeb Ingo Steinwart ODL 32 4 0 04 Nov 2020
A Unifying View on Implicit Bias in Training Linear Neural Networks Chulhee Yun Shankar Krishnan H. Mobahi MLT 125 82 0 06 Oct 2020