Batch Normalization Preconditioning for Neural Network Training

2 August 2021

Papers citing "Batch Normalization Preconditioning for Neural Network Training"

41 / 41 papers shown

Title
Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs Jonathan Frankle D. Schwab Ari S. Morcos 75 143 0 29 Feb 2020
Understanding and Improving Layer Normalization Jingjing Xu Xu Sun Zhiyuan Zhang Guangxiang Zhao Junyang Lin FAtt 85 354 0 16 Nov 2019
Four Things Everyone Should Know to Improve Batch Normalization Cecilia Summers M. Dinneen 55 52 0 09 Jun 2019
Batch Normalization is a Cause of Adversarial Vulnerability A. Galloway A. Golubeva T. Tanay M. Moussa Graham W. Taylor ODL AAML 63 80 0 06 May 2019
A Mean Field Theory of Batch Normalization Greg Yang Jeffrey Pennington Vinay Rao Jascha Narain Sohl-Dickstein S. Schoenholz 72 178 0 21 Feb 2019
A Simple Baseline for Bayesian Uncertainty in Deep Learning Wesley J. Maddox T. Garipov Pavel Izmailov Dmitry Vetrov A. Wilson BDL UQCV 84 808 0 07 Feb 2019
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization Sanjeev Arora Zhiyuan Li Kaifeng Lyu 80 131 0 10 Dec 2018
Revisit Batch Normalization: New Understanding from an Optimization View and a Refinement via Composition Optimization Xiangru Lian Ji Liu 28 6 0 15 Oct 2018
Preconditioner on Matrix Lie Group for SGD Xi-Lin Li 36 16 0 26 Sep 2018
Towards Understanding Regularization in Batch Normalization Ping Luo Xinjiang Wang Wenqi Shao Zhanglin Peng MLT AI4CE 63 180 0 04 Sep 2018
Laplacian Smoothing Gradient Descent Stanley Osher Bao Wang Penghang Yin Xiyang Luo Farzin Barekat Minh Pham A. Lin ODL 64 43 0 17 Jun 2018
Understanding Batch Normalization Johan Bjorck Carla P. Gomes B. Selman Kilian Q. Weinberger 150 610 0 01 Jun 2018
How Does Batch Normalization Help Optimization? Shibani Santurkar Dimitris Tsipras Andrew Ilyas Aleksander Madry ODL 97 1,542 0 29 May 2018
Group Normalization Yuxin Wu Kaiming He 231 3,660 0 22 Mar 2018
Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Sampling C. Riquelme George Tucker Jasper Snoek BDL 71 366 0 26 Feb 2018
$$\mathcal{G}$-SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space$ $\mathcal{G}$ -SGD: Optimizing ReLU Neural Networks in its Positively Scale-Invariant Space Qi Meng Shuxin Zheng Huishuai Zhang Wei Chen Zhi-Ming Ma Tie-Yan Liu 76 39 0 11 Feb 2018
Block-diagonal Hessian-free Optimization for Training Neural Networks Huishuai Zhang Caiming Xiong James Bradbury R. Socher ODL 30 22 0 20 Dec 2017
User-friendly guarantees for the Langevin Monte Carlo with inaccurate gradient A. Dalalyan Avetik G. Karagulyan 75 296 0 29 Sep 2017
Riemannian approach to batch normalization Minhyung Cho Jaehyung Lee 55 94 0 27 Sep 2017
Feedforward and Recurrent Neural Networks Backward Propagation and Hessian in Matrix Form Maxim Naumov 46 9 0 16 Sep 2017
L2 Regularization versus Batch and Weight Normalization Twan van Laarhoven 65 301 0 16 Jun 2017
On Calibration of Modern Neural Networks Chuan Guo Geoff Pleiss Yu Sun Kilian Q. Weinberger UQCV 299 5,833 0 14 Jun 2017
Sharp Minima Can Generalize For Deep Nets Laurent Dinh Razvan Pascanu Samy Bengio Yoshua Bengio ODL 116 772 0 15 Mar 2017
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models Sergey Ioffe BDL 61 540 0 10 Feb 2017
Deep Pyramidal Residual Networks Dongyoon Han Jiwhan Kim Junmo Kim 95 694 0 10 Oct 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 424 2,941 0 15 Sep 2016
Instance Normalization: The Missing Ingredient for Fast Stylization Dmitry Ulyanov Andrea Vedaldi Victor Lempitsky OOD 174 3,708 0 27 Jul 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 413 10,494 0 21 Jul 2016
Recurrent Batch Normalization Tim Cooijmans Nicolas Ballas César Laurent Çağlar Gülçehre Aaron Courville ODL 48 410 0 30 Mar 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 354 10,184 0 16 Mar 2016
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Tim Salimans Diederik P. Kingma ODL 194 1,942 0 25 Feb 2016
Stochastic Quasi-Newton Langevin Monte Carlo Umut Simsekli Roland Badeau A. Cemgil G. Richard BDL 52 62 0 10 Feb 2016
Preconditioned Stochastic Gradient Langevin Dynamics for Deep Neural Networks Chunyuan Li Changyou Chen David Carlson Lawrence Carin ODL BDL 88 325 0 23 Dec 2015
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,020 0 10 Dec 2015
Batch Normalized Recurrent Neural Networks César Laurent Gabriel Pereyra Philemon Brakel Yanzhe Zhang Yoshua Bengio 63 214 0 05 Oct 2015
Natural Neural Networks Guillaume Desjardins Karen Simonyan Razvan Pascanu Koray Kavukcuoglu 107 176 0 01 Jul 2015
Path-SGD: Path-Normalized Optimization in Deep Neural Networks Behnam Neyshabur Ruslan Salakhutdinov Nathan Srebro ODL 86 309 0 08 Jun 2015
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 104 1,014 0 19 Mar 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 463 43,305 0 11 Feb 2015
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,547 0 01 Sep 2014
Bayesian Posterior Sampling via Stochastic Gradient Fisher Scoring S. Ahn Anoop Korattikara Balan Max Welling 77 306 0 27 Jun 2012