High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation

3 May 2022

Jimmy Ba

Papers citing "High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation"

48 / 98 papers shown

Title
A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models Namjoon Suh Guang Cheng MedIm 30 12 0 14 Jan 2024
Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption Vasilii Feofanov Malik Tiomoko Aladin Virmaux 31 5 0 20 Oct 2023
Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks Mufan Bill Li Mihai Nica 28 2 0 18 Oct 2023
A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks Behrad Moniri Donghwan Lee Hamed Hassani Yan Sun MLT 40 19 0 11 Oct 2023
Grokking as the Transition from Lazy to Rich Training Dynamics Tanishq Kumar Blake Bordelon Samuel Gershman C. Pehlevan 35 31 0 09 Oct 2023
SGD Finds then Tunes Features in Two-Layer Neural Networks with near-Optimal Sample Complexity: A Case Study in the XOR problem Margalit Glasgow MLT 79 13 0 26 Sep 2023
Gradient-Based Feature Learning under Structured Data Alireza Mousavi-Hosseini Denny Wu Taiji Suzuki Murat A. Erdogdu MLT 37 18 0 07 Sep 2023
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 48 8 0 07 Sep 2023
Mechanism of feature learning in convolutional neural networks Daniel Beaglehole Adityanarayanan Radhakrishnan Parthe Pandit Misha Belkin FAtt MLT 34 14 0 01 Sep 2023
Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks Liam Collins Hamed Hassani Mahdi Soltanolkotabi Aryan Mokhtari Sanjay Shakkottai 39 10 0 13 Jul 2023
Quantitative CLTs in Deep Neural Networks Stefano Favaro Boris Hanin Domenico Marinucci I. Nourdin G. Peccati BDL 28 11 0 12 Jul 2023
How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model Francesco Cagnetta Leonardo Petrini Umberto M. Tomasini Alessandro Favero M. Wyart BDL 30 22 0 05 Jul 2023
Neural Hilbert Ladders: Multi-Layer Neural Networks in Function Space Zhengdao Chen 44 1 0 03 Jul 2023
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit Lorenzo Noci Chuning Li Mufan Bill Li Bobby He Thomas Hofmann Chris J. Maddison Daniel M. Roy 33 29 0 30 Jun 2023
Gaussian random field approximation via Stein's method with applications to wide random neural networks Krishnakumar Balasubramanian L. Goldstein Nathan Ross Adil Salim 30 8 0 28 Jun 2023
Graph Neural Networks Provably Benefit from Structural Information: A Feature Learning Perspective Wei Huang Yuanbin Cao Hong Wang Xin Cao Taiji Suzuki MLT 40 7 0 24 Jun 2023
On the Joint Interaction of Models, Data, and Features Yiding Jiang Christina Baek J. Zico Kolter FedML 28 4 0 07 Jun 2023
Birth of a Transformer: A Memory Viewpoint A. Bietti Vivien A. Cabannes Diane Bouchacourt Hervé Jégou Léon Bottou 35 82 0 01 Jun 2023
How Two-Layer Neural Networks Learn, One (Giant) Step at a Time Yatin Dandi Florent Krzakala Bruno Loureiro Luca Pesce Ludovic Stephan MLT 34 26 0 29 May 2023
Tight conditions for when the NTK approximation is valid Enric Boix-Adserà Etai Littwin 30 0 0 22 May 2023
Smoothing the Landscape Boosts the Signal for SGD: Optimal Sample Complexity for Learning Single Index Models Alexandru Damian Eshaan Nichani Rong Ge Jason D. Lee MLT 42 33 0 18 May 2023
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks Eshaan Nichani Alexandru Damian Jason D. Lee MLT 41 13 0 11 May 2023
High-dimensional scaling limits and fluctuations of online least-squares SGD with smooth covariance Krishnakumar Balasubramanian Promit Ghosal Ye He 38 5 0 03 Apr 2023
Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems Atsushi Nitanda Kazusato Oko Denny Wu Nobuhito Takenouchi Taiji Suzuki 32 3 0 06 Mar 2023
Learning time-scales in two-layers neural networks Raphael Berthier Andrea Montanari Kangjie Zhou 38 33 0 28 Feb 2023
SGD learning on neural networks: leap complexity and saddle-to-saddle dynamics Emmanuel Abbe Enric Boix-Adserà Theodor Misiakiewicz FedML MLT 79 73 0 21 Feb 2023
Are Gaussian data all you need? Extents and limits of universality in high-dimensional generalized linear estimation Luca Pesce Florent Krzakala Bruno Loureiro Ludovic Stephan 21 26 0 17 Feb 2023
Demystifying Disagreement-on-the-Line in High Dimensions Dong-Hwan Lee Behrad Moniri Xinmeng Huang Yan Sun Hamed Hassani 21 8 0 31 Jan 2023
Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn features Adityanarayanan Radhakrishnan Daniel Beaglehole Parthe Pandit M. Belkin FAtt MLT 31 11 0 28 Dec 2022
The Onset of Variance-Limited Behavior for Networks in the Lazy and Rich Regimes Alexander B. Atanasov Blake Bordelon Sabarish Sainathan C. Pehlevan 22 26 0 23 Dec 2022
Learning threshold neurons via the "edge of stability" Kwangjun Ahn Sébastien Bubeck Sinho Chewi Y. Lee Felipe Suarez Yi Zhang MLT 38 36 0 14 Dec 2022
On the symmetries in the dynamics of wide two-layer neural networks Karl Hajjar Lénaïc Chizat 13 11 0 16 Nov 2022
Spectral Evolution and Invariance in Linear-width Neural Networks Zhichao Wang A. Engel Anand D. Sarwate Ioana Dumitriu Tony Chiang 40 14 0 11 Nov 2022
Overparameterized random feature regression with nearly orthogonal data Zhichao Wang Yizhe Zhu 26 3 0 11 Nov 2022
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks Zhengdao Chen Eric Vanden-Eijnden Joan Bruna MLT 25 5 0 28 Oct 2022
Learning Single-Index Models with Shallow Neural Networks A. Bietti Joan Bruna Clayton Sanford M. Song 167 67 0 27 Oct 2022
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD Alireza Mousavi-Hosseini Sejun Park M. Girotti Ioannis Mitliagkas Murat A. Erdogdu MLT 324 48 0 29 Sep 2022
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit Boaz Barak Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 36 123 0 18 Jul 2022
The Neural Covariance SDE: Shaped Infinite Depth-and-Width Networks at Initialization Mufan Bill Li Mihai Nica Daniel M. Roy 35 36 0 06 Jun 2022
Dataset Distillation using Neural Feature Regression Yongchao Zhou E. Nezhadarya Jimmy Ba DD FedML 41 149 0 01 Jun 2022
Optimal Activation Functions for the Random Features Regression Model Jianxin Wang José Bento 34 3 0 31 May 2022
Excess Risk of Two-Layer ReLU Neural Networks in Teacher-Student Settings and its Superiority to Kernel Methods Shunta Akiyama Taiji Suzuki 30 6 0 30 May 2022
Gaussian Universality of Perceptrons with Random Labels Federica Gerace Florent Krzakala Bruno Loureiro Ludovic Stephan Lenka Zdeborová 51 23 0 26 May 2022
Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curves David Bosch Ashkan Panahi Ayça Özçelikkale Devdatt Dubhash MLT 21 2 0 06 Apr 2022
Random Feature Amplification: Feature Learning and Generalization in Neural Networks Spencer Frei Niladri S. Chatterji Peter L. Bartlett MLT 30 29 0 15 Feb 2022
Convex Analysis of the Mean Field Langevin Dynamics Atsushi Nitanda Denny Wu Taiji Suzuki MLT 68 64 0 25 Jan 2022
The Eigenlearning Framework: A Conservation Law Perspective on Kernel Regression and Wide Neural Networks James B. Simon Madeline Dickens Dhruva Karkada M. DeWeese 48 27 0 08 Oct 2021
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 159 234 0 04 Mar 2020