On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent

19 February 2021

Papers citing "On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descent"

50 / 53 papers shown

Title
Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias? Tom Jacobs Chao Zhou R. Burkholz OffRL AI4CE 33 0 0 17 Apr 2025
On the Cone Effect in the Learning Dynamics Zhanpeng Zhou Yongyi Yang Jie Ren Mahito Sugiyama Junchi Yan 53 0 0 20 Mar 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 38 0 0 28 Feb 2025
Optimization Insights into Deep Diagonal Linear Networks Hippolyte Labarrière C. Molinari Lorenzo Rosasco S. Villa Cristian Vega 76 0 0 21 Dec 2024
Slowing Down Forgetting in Continual Learning Pascal Janetzky Tobias Schlagenhauf Stefan Feuerriegel CLL 34 0 0 11 Nov 2024
A Mirror Descent Perspective of Smoothed Sign Descent Shuyang Wang Diego Klabjan 38 0 0 18 Oct 2024
Fast Training of Sinusoidal Neural Fields via Scaling Initialization Taesun Yeom Sangyoon Lee Jaeho Lee 53 2 0 07 Oct 2024
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 35 3 0 22 Sep 2024
Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learning Nadav Cohen Noam Razin 35 0 0 25 Aug 2024
Implicit Bias of Mirror Flow on Separable Data Scott Pesme Radu-Alexandru Dragomir Nicolas Flammarion 34 1 0 18 Jun 2024
Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning D. Kunin Allan Raventós Clémentine Dominé Feng Chen David Klindt Andrew M. Saxe Surya Ganguli MLT 45 15 0 10 Jun 2024
Implicit Regularization of Gradient Flow on One-Layer Softmax Attention Heejune Sheen Siyu Chen Tianhao Wang Harrison H. Zhou MLT 35 10 0 13 Mar 2024
Gradient Descent with Polyak's Momentum Finds Flatter Minima via Large Catapults Prin Phunyaphibarn Junghyun Lee Bohan Wang Huishuai Zhang Chulhee Yun 21 0 0 25 Nov 2023
How connectivity structure shapes rich and lazy learning in neural circuits Yuhan Helena Liu A. Baratin Jonathan H. Cornford Stefan Mihalas E. Shea-Brown Guillaume Lajoie 40 14 0 12 Oct 2023
A Theoretical Analysis of Noise Geometry in Stochastic Gradient Descent Mingze Wang Lei Wu 22 3 0 01 Oct 2023
Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics Yehonatan Avidan Qianyi Li H. Sompolinsky 60 8 0 08 Sep 2023
The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning Nikhil Ghosh Spencer Frei Wooseok Ha Ting Yu MLT 32 3 0 06 Aug 2023
Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networks J. S. Wind Vegard Antun A. Hansen 19 4 0 13 Jul 2023
Abide by the Law and Follow the Flow: Conservation Laws for Gradient Flows Sibylle Marcotte Rémi Gribonval Gabriel Peyré 30 16 0 30 Jun 2023
The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks Mor Shpigel Nacson Rotem Mulayoff Greg Ongie T. Michaeli Daniel Soudry 20 12 0 30 Jun 2023
Trained Transformers Learn Linear Models In-Context Ruiqi Zhang Spencer Frei Peter L. Bartlett 26 173 0 16 Jun 2023
Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputs D. Chistikov Matthias Englert R. Lazic MLT 36 12 0 10 Jun 2023
Combining Explicit and Implicit Regularization for Efficient Learning in Deep Networks Dan Zhao 14 5 0 01 Jun 2023
Gradient Descent Monotonically Decreases the Sharpness of Gradient Flow Solutions in Scalar Networks and Beyond Itai Kreisler Mor Shpigel Nacson Daniel Soudry Y. Carmon 25 13 0 22 May 2023
Saddle-to-Saddle Dynamics in Diagonal Linear Networks Scott Pesme Nicolas Flammarion 31 35 0 02 Apr 2023
(S)GD over Diagonal Linear Networks: Implicit Regularisation, Large Stepsizes and Edge of Stability Mathieu Even Scott Pesme Suriya Gunasekar Nicolas Flammarion 28 16 0 17 Feb 2023
Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regression Mo Zhou Rong Ge 27 2 0 01 Feb 2023
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 40 49 0 25 Oct 2022
Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets Edo Cohen-Karlik Itamar Menuhin-Gruman Raja Giryes Nadav Cohen Amir Globerson 25 4 0 25 Oct 2022
From Gradient Flow on Population Loss to Learning with Stochastic Gradient Descent Satyen Kale Jason D. Lee Chris De Sa Ayush Sekhari Karthik Sridharan 24 4 0 13 Oct 2022
The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima Peter L. Bartlett Philip M. Long Olivier Bousquet 73 34 0 04 Oct 2022
Deep Linear Networks can Benignly Overfit when Shallow Ones Do Niladri S. Chatterji Philip M. Long 20 8 0 19 Sep 2022
Incremental Learning in Diagonal Linear Networks Raphael Berthier CLL AI4CE 30 16 0 31 Aug 2022
On the Implicit Bias in Deep-Learning Algorithms Gal Vardi FedML AI4CE 34 72 0 26 Aug 2022
Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descent Zhiyuan Li Tianhao Wang Jason D. Lee Sanjeev Arora 34 27 0 08 Jul 2022
The alignment property of SGD noise and how it helps select flat minima: A stability analysis Lei Wu Mingze Wang Weijie Su MLT 22 31 0 06 Jul 2022
Reconstructing Training Data from Trained Neural Networks Niv Haim Gal Vardi Gilad Yehudai Ohad Shamir Michal Irani 40 132 0 15 Jun 2022
Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embedding Tianyang Hu Zhili Liu Fengwei Zhou Wenjia Wang Weiran Huang SSL 41 26 0 30 May 2022
Smooth over-parameterized solvers for non-smooth structured optimization C. Poon Gabriel Peyré 24 18 0 03 May 2022
Implicit Regularization Properties of Variance Reduced Stochastic Mirror Descent Yiling Luo X. Huo Y. Mei 23 1 0 29 Apr 2022
Support Vectors and Gradient Dynamics of Single-Neuron ReLU Networks Sangmin Lee Byeongsu Sim Jong Chul Ye MLT 24 0 0 11 Feb 2022
Implicit Regularization Towards Rank Minimization in ReLU Networks Nadav Timor Gal Vardi Ohad Shamir 26 49 0 30 Jan 2022
Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks Noam Razin Asaf Maman Nadav Cohen 43 29 0 27 Jan 2022
More is Less: Inducing Sparsity via Overparameterization H. Chou J. Maly Holger Rauhut 30 25 0 21 Dec 2021
What Happens after SGD Reaches Zero Loss? --A Mathematical Framework Zhiyuan Li Tianhao Wang Sanjeev Arora MLT 90 98 0 13 Oct 2021
Foolish Crowds Support Benign Overfitting Niladri S. Chatterji Philip M. Long 83 20 0 06 Oct 2021
On Margin Maximization in Linear and ReLU Networks Gal Vardi Ohad Shamir Nathan Srebro 50 28 0 06 Oct 2021
Continuous vs. Discrete Optimization of Deep Neural Networks Omer Elkabetz Nadav Cohen 65 44 0 14 Jul 2021
A Theoretical Analysis of Fine-tuning with Linear Teachers Gal Shachaf Alon Brutzkus Amir Globerson 28 17 0 04 Jul 2021
Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity Scott Pesme Loucas Pillaud-Vivien Nicolas Flammarion 27 98 0 17 Jun 2021