Gradient flow dynamics of shallow ReLU networks for square loss and
orthogonal inputs

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

2 June 2022

Etienne Boursier

Loucas Pillaud-Vivien

Nicolas Flammarion

Papers citing "Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs"

19 / 19 papers shown

Title
A distributional simplicity bias in the learning dynamics of transformers Riccardo Rende Federica Gerace A. Laio Sebastian Goldt 79 8 0 17 Feb 2025
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence Berfin Simsek Amire Bendjeddou Daniel Hsu 44 0 0 13 Nov 2024
Early Directional Convergence in Deep Homogeneous Neural Networks for Small Initializations Akshay Kumar Jarvis Haupt ODL 44 3 0 12 Mar 2024
Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding Zhengqing Wu Berfin Simsek Francois Ged ODL 45 0 0 08 Feb 2024
Early Neuron Alignment in Two-layer ReLU Networks with Small Initialization Hancheng Min Enrique Mallada René Vidal MLT 34 18 0 24 Jul 2023
Saddle-to-Saddle Dynamics in Diagonal Linear Networks Scott Pesme Nicolas Flammarion 31 35 0 02 Apr 2023
Penalising the biases in norm regularisation enforces sparsity Etienne Boursier Nicolas Flammarion 37 14 0 02 Mar 2023
Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuron Weihang Xu S. Du 37 16 0 20 Feb 2023
From high-dimensional & mean-field dynamics to dimensionless ODEs: A unifying approach to SGD in two-layers networks Luca Arnaboldi Ludovic Stephan Florent Krzakala Bruno Loureiro MLT 38 31 0 12 Feb 2023
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing Jikai Jin Zhiyuan Li Kaifeng Lyu S. Du Jason D. Lee MLT 54 34 0 27 Jan 2023
Regression as Classification: Influence of Task Formulation on Neural Network Features Lawrence Stewart Francis R. Bach Quentin Berthet Jean-Philippe Vert 29 24 0 10 Nov 2022
Proximal Mean Field Learning in Shallow Neural Networks Alexis M. H. Teter Iman Nodozi A. Halder FedML 43 1 0 25 Oct 2022
SGD with Large Step Sizes Learns Sparse Features Maksym Andriushchenko Aditya Varre Loucas Pillaud-Vivien Nicolas Flammarion 45 56 0 11 Oct 2022
Magnitude and Angle Dynamics in Training Single ReLU Neurons Sangmin Lee Byeongsu Sim Jong Chul Ye MLT 96 6 0 27 Sep 2022
Git Re-Basin: Merging Models modulo Permutation Symmetries Samuel K. Ainsworth J. Hayase S. Srinivasa MoMe 255 314 0 11 Sep 2022
On the Implicit Bias in Deep-Learning Algorithms Gal Vardi FedML AI4CE 34 72 0 26 Aug 2022
Random Feature Amplification: Feature Learning and Generalization in Neural Networks Spencer Frei Niladri S. Chatterji Peter L. Bartlett MLT 30 29 0 15 Feb 2022
A Local Convergence Theory for Mildly Over-Parameterized Two-Layer Neural Network Mo Zhou Rong Ge Chi Jin 74 44 0 04 Feb 2021
Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach Grant M. Rotskoff Eric Vanden-Eijnden 59 118 0 02 May 2018