Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks

4 March 2016

Devansh Arpit

Yingbo Zhou

Bhargava U. Kota

V. Govindaraju

ArXiv PDF HTML

Papers citing "Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks"

27 / 27 papers shown

Title
Normalization and effective learning rates in reinforcement learning Clare Lyle Zeyu Zheng Khimya Khetarpal James Martens H. V. Hasselt Razvan Pascanu Will Dabney 19 7 0 01 Jul 2024
Analyzing and Improving the Training Dynamics of Diffusion Models Tero Karras M. Aittala J. Lehtinen Janne Hellsten Timo Aila S. Laine 30 155 0 05 Dec 2023
BCN: Batch Channel Normalization for Image Classification Afifa Khaled Chao Li Jia Ning Kun He 15 6 0 01 Dec 2023
Information Geometrically Generalized Covariate Shift Adaptation Masanari Kimura H. Hino OOD 11 5 0 19 Apr 2023
Noise Injection as a Probe of Deep Learning Dynamics Noam Levi I. Bloch M. Freytsis T. Volansky 40 2 0 24 Oct 2022
Batch Layer Normalization, A new normalization layer for CNNs and RNN A. Ziaee Erion cCano 13 12 0 19 Sep 2022
Revisiting Batch Norm Initialization Jim Davis Logan Frank 22 4 0 26 Oct 2021
"BNN - BN = ?": Training Binary Neural Networks without Batch Normalization Tianlong Chen Zhenyu (Allen) Zhang Xu Ouyang Zechun Liu Zhiqiang Shen Zhangyang Wang MQ 37 36 0 16 Apr 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Momentum^2 Teacher: Momentum Teacher with Momentum Statistics for Self-Supervised Learning Zeming Li Songtao Liu Jian Sun 51 16 0 19 Jan 2021
A Comprehensive and Modularized Statistical Framework for Gradient Norm Equality in Deep Neural Networks Zhaodong Chen Lei Deng Bangyan Wang Guoqi Li Yuan Xie 32 28 0 01 Jan 2020
Spectral Regularization for Combating Mode Collapse in GANs Kanglin Liu Wenming Tang Fei Zhou Guoping Qiu GAN DRL 30 81 0 29 Aug 2019
EvalNorm: Estimating Batch Normalization Statistics for Evaluation Saurabh Singh Abhinav Shrivastava 24 51 0 12 Apr 2019
Micro-Batch Training with Batch-Channel Normalization and Weight Standardization Siyuan Qiao Huiyu Wang Chenxi Liu Wei Shen Alan Yuille MQ 29 144 0 25 Mar 2019
Accelerating Training of Deep Neural Networks with a Standardization Loss Jasmine Collins Johannes Ballé Jonathon Shlens 16 3 0 03 Mar 2019
Mode Normalization Lucas Deecke Iain Murray Hakan Bilen OOD 29 33 0 12 Oct 2018
NU-LiteNet: Mobile Landmark Recognition using Convolutional Neural Networks C. Termritthikun S. Kanprachar P. Muneesawang 14 20 0 02 Oct 2018
Revisiting Small Batch Training for Deep Neural Networks Dominic Masters Carlo Luschi ODL 23 658 0 20 Apr 2018
Spectral Normalization for Generative Adversarial Networks Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida ODL 17 4,395 0 16 Feb 2018
The exploding gradient problem demystified - definition, prevalence, impact, origin, tradeoffs, and solutions George Philipp D. Song J. Carbonell ODL 29 46 0 15 Dec 2017
Riemannian approach to batch normalization Minhyung Cho Jaehyung Lee 26 93 0 27 Sep 2017
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification Igor Gitman Boris Ginsburg 8 65 0 24 Sep 2017
Reducing Bias in Production Speech Models Eric Battenberg R. Child Adam Coates Christopher Fougner Yashesh Gaur ... Vinay Rao S. Satheesh David Seetapun Anuroop Sriram Zhenyao Zhu 30 10 0 11 May 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 46 755 0 15 Mar 2017
All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation Di Xie Jiang Xiong Shiliang Pu 19 181 0 06 Mar 2017
Cosine Normalization: Using Cosine Similarity Instead of Dot Product in Neural Networks Chunjie Luo Jianfeng Zhan Lei Wang Qiang Yang 19 198 0 20 Feb 2017
Adding Gradient Noise Improves Learning for Very Deep Networks Arvind Neelakantan Luke Vilnis Quoc V. Le Ilya Sutskever Lukasz Kaiser Karol Kurach James Martens AI4CE ODL 27 541 0 21 Nov 2015