How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers

19 October 2020

Yang You

Papers citing "How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers"

23 / 23 papers shown

Title
Using a thousand optimization tasks to learn hyperparameter search strategies Luke Metz Niru Maheswaranathan Ruoxi Sun C. Freeman Ben Poole Jascha Narain Sohl-Dickstein 55 46 0 27 Feb 2020
On Empirical Comparisons of Optimizers for Deep Learning Dami Choi Christopher J. Shallue Zachary Nado Jaehoon Lee Chris J. Maddison George E. Dahl 66 259 0 11 Oct 2019
On the Variance of the Adaptive Learning Rate and Beyond Liyuan Liu Haoming Jiang Pengcheng He Weizhu Chen Xiaodong Liu Jianfeng Gao Jiawei Han ODL 208 1,894 0 08 Aug 2019
Lookahead Optimizer: k steps forward, 1 step back Michael Ruogu Zhang James Lucas Geoffrey E. Hinton Jimmy Ba ODL 99 725 0 19 Jul 2019
Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks Wei-Lin Chiang Xuanqing Liu Si Si Yang Li Samy Bengio Cho-Jui Hsieh GNN 136 1,268 0 20 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 72 2,482 0 19 Apr 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 190 991 0 01 Apr 2019
The importance of better models in stochastic optimization Hilal Asi John C. Duchi 38 73 0 20 Mar 2019
DeepOBS: A Deep Learning Optimizer Benchmark Suite Frank Schneider Lukas Balles Philipp Hennig ODL 98 71 0 13 Mar 2019
Adaptive Gradient Methods with Dynamic Bound of Learning Rate Liangchen Luo Yuanhao Xiong Yan Liu Xu Sun ODL 53 600 0 26 Feb 2019
Minimum weight norm models do not always generalize well for over-parameterized problems Vatsal Shah Anastasios Kyrillidis Sujay Sanghavi 38 21 0 16 Nov 2018
Measuring the Effects of Data Parallelism on Neural Network Training Christopher J. Shallue Jaehoon Lee J. Antognini J. Mamou J. Ketterling Yao Wang 78 408 0 08 Nov 2018
BOHB: Robust and Efficient Hyperparameter Optimization at Scale Stefan Falkner Aaron Klein Frank Hutter BDL 177 1,077 0 04 Jul 2018
TBD: Benchmarking and Analyzing Deep Neural Network Training Hongyu Zhu Mohamed Akrout Bojian Zheng Andrew Pelegris Amar Phanishayee Bianca Schroeder Gennady Pekhimenko 51 80 0 16 Mar 2018
Spectral Normalization for Generative Adversarial Networks Takeru Miyato Toshiki Kataoka Masanori Koyama Yuichi Yoshida ODL 153 4,421 0 16 Feb 2018
Large Batch Training of Convolutional Networks Yang You Igor Gitman Boris Ginsburg ODL 116 844 0 13 Aug 2017
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 288 18,685 0 20 Jul 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 52 1,023 0 23 May 2017
OpenAI Gym Greg Brockman Vicki Cheung Ludwig Pettersson Jonas Schneider John Schulman Jie Tang Wojciech Zaremba OffRL ODL 188 5,056 0 05 Jun 2016
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization Lisha Li Kevin Jamieson Giulia DeSalvo Afshin Rostamizadeh Ameet Talwalkar 195 2,307 0 21 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.7K 192,638 0 10 Dec 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.3K 149,474 0 22 Dec 2014
Auto-Encoding Variational Bayes Diederik P. Kingma Max Welling BDL 397 16,962 0 20 Dec 2013