Exponential convergence rates for Batch Normalization: The power of
length-direction decoupling in non-convex optimization

Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization

27 May 2018

Hadi Daneshmand

Aurelien Lucchi

Papers citing "Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization"

18 / 18 papers shown

Title
AADNet: Exploring EEG Spatiotemporal Information for Fast and Accurate Orientation and Timbre Detection of Auditory Attention Based on A Cue-Masked Paradigm Keren Shi Xu Liu Xue Yuan Haijie Shang Ruiting Dai Hanbin Wang Yunfa Fu N. Jiang Jiayuan He 38 1 0 08 Jan 2025
Hidden Synergy: $L_1$ Weight Normalization and 1-Path-Norm Regularization Aditya Biswas 41 0 0 29 Apr 2024
The Implicit Bias of Batch Normalization in Linear Models and Two-layer Linear Convolutional Neural Networks Yuan Cao Difan Zou Yuan-Fang Li Quanquan Gu MLT 37 5 0 20 Jun 2023
Batch Normalization Explained Randall Balestriero Richard G. Baraniuk AAML 36 16 0 29 Sep 2022
Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction Kaifeng Lyu Zhiyuan Li Sanjeev Arora FAtt 45 71 0 14 Jun 2022
FedBN: Federated Learning on Non-IID Features via Local Batch Normalization Xiaoxiao Li Meirui Jiang Xiaofei Zhang Michael Kamp Qi Dou OOD FedML 168 790 0 15 Feb 2021
Group Whitening: Balancing Learning Efficiency and Representational Capacity Lei Huang Yi Zhou Li Liu Fan Zhu Ling Shao 33 21 0 28 Sep 2020
GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training Tianle Cai Shengjie Luo Keyulu Xu Di He Tie-Yan Liu Liwei Wang GNN 32 159 0 07 Sep 2020
DO-Conv: Depthwise Over-parameterized Convolutional Layer Jinming Cao Yangyan Li Mingchao Sun Ying-Cong Chen Dani Lischinski Daniel Cohen-Or Baoquan Chen Changhe Tu OOD 33 166 0 22 Jun 2020
New Interpretations of Normalization Methods in Deep Learning Jiacheng Sun Xiangyong Cao Hanwen Liang Weiran Huang Zewei Chen Zhenguo Li 21 35 0 16 Jun 2020
Optimization for deep learning: theory and algorithms Ruoyu Sun ODL 27 168 0 19 Dec 2019
How Does Learning Rate Decay Help Modern Neural Networks? Kaichao You Mingsheng Long Jianmin Wang Michael I. Jordan 30 4 0 05 Aug 2019
Principled Training of Neural Networks with Direct Feedback Alignment Julien Launay Iacopo Poli Florent Krzakala 24 35 0 11 Jun 2019
The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks Ryo Karakida S. Akaho S. Amari 27 40 0 07 Jun 2019
Unsupervised Domain Adaptation using Feature-Whitening and Consensus Loss Subhankar Roy Aliaksandr Siarohin E. Sangineto Samuel Rota Buló N. Sebe Elisa Ricci 39 163 0 07 Mar 2019
Accelerating Training of Deep Neural Networks with a Standardization Loss Jasmine Collins Johannes Ballé Jonathon Shlens 21 3 0 03 Mar 2019
Information Geometry of Orthogonal Initializations and Training Piotr A. Sokól Il-Su Park AI4CE 80 16 0 09 Oct 2018
How Does Batch Normalization Help Optimization? Shibani Santurkar Dimitris Tsipras Andrew Ilyas A. Madry ODL 32 1,523 0 29 May 2018