Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient

16 December 2020

Papers citing "Study on the Large Batch Size Training of Neural Networks Based on the Second Order Gradient"

24 / 24 papers shown

Title
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks Yuanzhi Li Colin Wei Tengyu Ma 47 299 0 10 Jul 2019
Gradient Noise Convolution (GNC): Smoothing Loss Function for Distributed Large-Batch SGD Kosuke Haruki Taiji Suzuki Yohei Hamakawa Takeshi Toda Ryuji Sakai M. Ozawa Mitsuhiro Kimura ODL 46 17 0 26 Jun 2019
Large-Batch Training for LSTM and Beyond Yang You Jonathan Hseu Chris Ying J. Demmel Kurt Keutzer Cho-Jui Hsieh 52 90 0 24 Jan 2019
An Empirical Model of Large-Batch Training Sam McCandlish Jared Kaplan Dario Amodei OpenAI Dota Team 65 277 0 14 Dec 2018
Nonlinear Conjugate Gradients For Scaling Synchronous Distributed DNN Training Saurabh N. Adya Vinay Palakkode Oncel Tuzel 16 4 0 07 Dec 2018
Towards Theoretical Understanding of Large Batch Training in Stochastic Gradient Descent Xiaowu Dai Yuhua Zhu 51 11 0 03 Dec 2018
Massively Distributed SGD: ImageNet/ResNet-50 Training in a Flash Hiroaki Mikami Hisahiro Suganuma Pongsakorn U-chupala Yoshiki Tanaka Yuichi Kageyama 50 77 0 13 Nov 2018
Large batch size training of neural networks with adversarial training and second-order information Z. Yao A. Gholami Daiyaan Arfeen Richard Liaw Joseph E. Gonzalez Kurt Keutzer Michael W. Mahoney ODL 68 42 0 02 Oct 2018
Don't Use Large Mini-Batches, Use Local SGD Tao R. Lin Sebastian U. Stich Kumar Kshitij Patel Martin Jaggi 117 432 0 22 Aug 2018
SmoothOut: Smoothing Out Sharp Minima to Improve Generalization in Deep Learning W. Wen Yandan Wang Feng Yan Cong Xu Chunpeng Wu Yiran Chen H. Li 61 51 0 21 May 2018
AdaBatch: Adaptive Batch Sizes for Training Deep Neural Networks Aditya Devarakonda Maxim Naumov M. Garland ODL 73 136 0 06 Dec 2017
MegDet: A Large Mini-Batch Object Detector Chao Peng Tete Xiao Zeming Li Yuning Jiang Xiangyu Zhang Kai Jia Gang Yu Jian Sun ObjD 179 318 0 20 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 99 996 0 01 Nov 2017
Proportionate gradient updates with PercentDelta Sami Abu-El-Haija 54 7 0 24 Aug 2017
Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates L. Smith Nicholay Topin AI4CE 84 520 0 23 Aug 2017
Large Batch Training of Convolutional Networks Yang You Igor Gitman Boris Ginsburg ODL 128 848 0 13 Aug 2017
Exploring Generalization in Deep Learning Behnam Neyshabur Srinadh Bhojanapalli David A. McAllester Nathan Srebro FAtt 148 1,256 0 27 Jun 2017
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour Priya Goyal Piotr Dollár Ross B. Girshick P. Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He 3DH 126 3,681 0 08 Jun 2017
Train longer, generalize better: closing the generalization gap in large batch training of neural networks Elad Hoffer Itay Hubara Daniel Soudry ODL 173 799 0 24 May 2017
On the diffusion approximation of nonconvex stochastic gradient descent Junyang Qian C. J. Li Lei Li Jianguo Liu DiffM 51 24 0 22 May 2017
Distributed Training of Deep Neural Networks: Theoretical and Practical Limits of Parallel Scalability J. Keuper Franz-Josef Pfreundt GNN 83 98 0 22 Sep 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 424 2,937 0 15 Sep 2016
The Effects of Hyperparameters on SGD Training of Neural Networks Thomas Breuel 117 63 0 12 Aug 2015
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 101 1,014 0 19 Mar 2015