Optimization Theory for ReLU Neural Networks Trained with Normalization
Layers

Optimization Theory for ReLU Neural Networks Trained with Normalization Layers

11 June 2020

Quanquan Gu

Guido Montúfar

Papers citing "Optimization Theory for ReLU Neural Networks Trained with Normalization Layers"

7 / 7 papers shown

Title
Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization Aditya Biswas 41 0 0 29 Apr 2024
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks Yuan Cao Difan Zou Yuan-Fang Li Quanquan Gu MLT 37 5 0 20 Jun 2023
Robust Implicit Regularization via Weight Normalization H. Chou Holger Rauhut Rachel A. Ward 40 7 0 09 May 2023
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 52 71 0 14 Jun 2022
Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks Benjamin Bowman Guido Montúfar 28 11 0 12 Jan 2022
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization Xiaoxiao Li Meirui Jiang Xiaofei Zhang Michael Kamp Qi Dou OOD FedML 168 791 0 15 Feb 2021
A Dynamical View on Optimization Algorithms of Overparameterized Neural Networks Zhiqi Bu Shiyun Xu Kan Chen 38 17 0 25 Oct 2020