Get rich quick: exact solutions reveal how unbalanced initializations
promote rapid feature learning

Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning

10 June 2024

Allan Raventós

Clémentine Dominé

Feng Chen

Papers citing "Get rich quick: exact solutions reveal how unbalanced initializations promote rapid feature learning"

18 / 18 papers shown

Title
An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Research Patrik Reizinger Randall Balestriero David Klindt Wieland Brendel 40 0 0 17 Apr 2025
Beyond the Next Token: Towards Prompt-Robust Zero-Shot Classification via Efficient Multi-Token Prediction Junlang Qian Zixiao Zhu Hanzhang Zhou Zijian Feng Zepeng Zhai K. Mao AAML VLM 38 0 0 04 Apr 2025
Dynamically Learning to Integrate in Recurrent Neural Networks Blake Bordelon Jordan Cotler C. Pehlevan Jacob A. Zavatone-Veth 53 2 0 24 Mar 2025
On the Cone Effect in the Learning Dynamics Zhanpeng Zhou Yongyi Yang Jie Ren Mahito Sugiyama Junchi Yan 53 0 0 20 Mar 2025
The Spectral Bias of Shallow Neural Network Learning is Shaped by the Choice of Non-linearity Justin Sahs Ryan Pyle Fabio Anselmi Ankit B. Patel 50 0 0 13 Mar 2025
A Theory of Initialisation's Impact on Specialisation Devon Jarvis Sebastian Lee Clémentine Dominé Andrew M. Saxe Stefano Sarao Mannelli CLL 67 2 0 04 Mar 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 38 0 0 28 Feb 2025
Deep Linear Network Training Dynamics from Random Initialization: Data, Width, Depth, and Hyperparameter Transfer Blake Bordelon C. Pehlevan AI4CE 59 1 0 04 Feb 2025
Plastic Learning with Deep Fourier Features Alex Lewandowski Dale Schuurmans Marlos C. Machado CLL 42 3 0 27 Oct 2024
Features are fate: a theory of transfer learning in high-dimensional regression Javan Tahir Surya Ganguli Grant M. Rotskoff 32 1 0 10 Oct 2024
Fast Training of Sinusoidal Neural Fields via Scaling Initialization Taesun Yeom Sangyoon Lee Jaeho Lee 53 2 0 07 Oct 2024
How Feature Learning Can Improve Neural Scaling Laws Blake Bordelon Alexander B. Atanasov C. Pehlevan 51 12 0 26 Sep 2024
From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks Clémentine Dominé Nicolas Anguita A. Proca Lukas Braun D. Kunin P. Mediano Andrew M. Saxe 30 3 0 22 Sep 2024
Towards understanding epoch-wise double descent in two-layer linear neural networks Amanda Olmin Fredrik Lindsten MLT 27 3 0 13 Jul 2024
Asymptotics of feature learning in two-layer networks after one gradient-step Hugo Cui Luca Pesce Yatin Dandi Florent Krzakala Yue M. Lu Lenka Zdeborová Bruno Loureiro MLT 44 16 0 07 Feb 2024
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 99 77 0 08 Dec 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 159 234 0 04 Mar 2020
Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach Grant M. Rotskoff Eric Vanden-Eijnden 59 118 0 02 May 2018