Norm matters: efficient and accurate normalization schemes in deep networks

5 March 2018

Papers citing "Norm matters: efficient and accurate normalization schemes in deep networks"

32 / 32 papers shown

Title
Group Normalization Yuxin Wu Kaiming He 147 3,626 0 22 Mar 2018
L1-Norm Batch Normalization for Efficient Training of Deep Neural Networks Shuang Wu Guoqi Li Lei Deng Liu Liu Yuan Xie Luping Shi 37 117 0 27 Feb 2018
Spectral Normalization for Generative Adversarial Networks Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida ODL 141 4,421 0 16 Feb 2018
Mixed Precision Training of Convolutional Neural Networks using Integer Operations Dipankar Das Naveen Mellempudi Dheevatsa Mudigere Dhiraj D. Kalamkar Sasikanth Avancha ... J. Corbal N. Shustrov R. Dubtsov Evarist Fomenko V. Pirogov MQ 61 154 0 03 Feb 2018
In-Place Activated BatchNorm for Memory-Optimized Training of DNNs Samuel Rota Buló Lorenzo Porzi Peter Kontschieder 47 356 0 07 Dec 2017
Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks Urs Koster T. Webb Xin Eric Wang Marcel Nassar Arjun K. Bansal ... Luke Hornof A. Khosrowshahi Carey Kloss Ruby J. Pai N. Rao MQ 40 261 0 06 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 93 990 0 01 Nov 2017
The Implicit Bias of Gradient Descent on Separable Data Daniel Soudry Elad Hoffer Mor Shpigel Nacson Suriya Gunasekar Nathan Srebro 83 908 0 27 Oct 2017
Mixed Precision Training Paulius Micikevicius Sharan Narang Jonah Alben G. Diamos Erich Elsen ... Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh Hao Wu 139 1,779 0 10 Oct 2017
Projection Based Weight Normalization for Deep Neural Networks Lei Huang Xianglong Liu B. Lang Yue Liu 44 18 0 06 Oct 2017
Comparison of Batch Normalization and Weight Normalization Algorithms for the Large-scale Image Classification Igor Gitman Boris Ginsburg 32 65 0 24 Sep 2017
Shifting Mean Activation Towards Zero with Bipolar Activation Functions L. Eidnes Arild Nøkland 33 18 0 12 Sep 2017
L2 Regularization versus Batch and Weight Normalization Twan van Laarhoven 38 295 0 16 Jun 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 490 129,831 0 12 Jun 2017
Self-Normalizing Neural Networks Günter Klambauer Thomas Unterthiner Andreas Mayr Sepp Hochreiter 249 2,496 0 08 Jun 2017
Train longer, generalize better: closing the generalization gap in large batch training of neural networks Elad Hoffer Itay Hubara Daniel Soudry ODL 144 799 0 24 May 2017
On the Effects of Batch and Weight Normalization in Generative Adversarial Networks Sitao Xiang Hao Li GAN 54 83 0 13 Apr 2017
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models Sergey Ioffe BDL 45 538 0 10 Feb 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 271 4,620 0 10 Nov 2016
Accelerating Deep Convolutional Networks using low-precision and sparsity Ganesh Venkatesh Eriko Nurvitadhi Debbie Marr 58 135 0 02 Oct 2016
Instance Normalization: The Missing Ingredient for Fast Stylization Dmitry Ulyanov Andrea Vedaldi Victor Lempitsky OOD 127 3,689 0 27 Jul 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 285 10,412 0 21 Jul 2016
Improved Techniques for Training GANs Tim Salimans Ian Goodfellow Wojciech Zaremba Vicki Cheung Alec Radford Xi Chen GAN 383 8,999 0 10 Jun 2016
Recurrent Batch Normalization Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville ODL 37 410 0 30 Mar 2016
Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks Devansh Arpit Yingbo Zhou Bhargava U. Kota V. Govindaraju 43 127 0 04 Mar 2016
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Tim Salimans Diederik P. Kingma ODL 148 1,933 0 25 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.4K 192,638 0 10 Dec 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 354 43,154 0 11 Feb 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 203 18,534 0 06 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 902 149,474 0 22 Dec 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 963 99,991 0 04 Sep 2014
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 397 27,205 0 01 Sep 2014