v1v2v3 (latest)

On Empirical Comparisons of Optimizers for Deep Learning

11 October 2019

Dami Choi

Christopher J. Shallue

Papers citing "On Empirical Comparisons of Optimizers for Deep Learning"

50 / 105 papers shown

Title
Nearest neighbor search with compact codes: A decoder perspective Kenza Amara Matthijs Douze Alexandre Sablayrolles Hervé Jégou MQ 118 6 0 17 Dec 2021
Predicting the utility of search spaces for black-box optimization: a simple, budget-aware approach Setareh Ariafar Justin Gilmer Zachary Nado Jasper Snoek Rodolphe Jenatton George E. Dahl 70 2 0 15 Dec 2021
Towards Robust and Automatic Hyper-Parameter Tunning Mathieu Tuli Mahdi S. Hosseini Konstantinos N. Plataniotis 120 1 0 28 Nov 2021
Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation David Acuna Jonah Philion Sanja Fidler OOD 81 16 0 15 Nov 2021
Large-Scale Deep Learning Optimizations: A Comprehensive Survey Xiaoxin He Fuzhao Xue Xiaozhe Ren Yang You 90 15 0 01 Nov 2021
Sustainable AI: Environmental Implications, Challenges and Opportunities Carole-Jean Wu Ramya Raghavendra Udit Gupta Bilge Acun Newsha Ardalani ... Maximilian Balandat Joe Spisak R. Jain Michael G. Rabbat K. Hazelwood 159 418 0 30 Oct 2021
Efficient Meta Subspace Optimization Yoni Choukroun Michael Katz 85 1 0 28 Oct 2021
Stable Anderson Acceleration for Deep Learning Massimiliano Lupo Pasini Junqi Yin Viktor Reshniak M. Stoyanov 59 4 0 26 Oct 2021
A Loss Curvature Perspective on Training Instability in Deep Learning Justin Gilmer Behrooz Ghorbani Ankush Garg Sneha Kudugunta Behnam Neyshabur David E. Cardoze George E. Dahl Zachary Nado Orhan Firat ODL 77 37 0 08 Oct 2021
Pre-trained Gaussian processes for Bayesian optimization Zehao Wang George E. Dahl Kevin Swersky Chansoo Lee Zachary Nado Justin Gilmer Jasper Snoek Zoubin Ghahramani 151 46 0 16 Sep 2021
Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization Difan Zou Yuan Cao Yuanzhi Li Quanquan Gu MLT AI4CE 113 44 0 25 Aug 2021
Logit Attenuating Weight Normalization Aman Gupta R. Ramanath Jun Shi Anika Ramachandran Sirou Zhou Mingzhou Zhou S. Keerthi 75 1 0 12 Aug 2021
Don't Sweep your Learning Rate under the Rug: A Closer Look at Cross-modal Transfer of Pretrained Transformers Dan Rothermel Margaret Li Tim Rocktaschel Jakob N. Foerster 72 6 0 26 Jul 2021
A Shallow Ritz Method for Elliptic Problems with Singular Sources M. Lai Che-Chia Chang Wei-Syuan Lin Wei-Fan Hu Te-Sheng Lin 72 14 0 26 Jul 2021
L2M: Practical posterior Laplace approximation with optimization-driven second moment estimation C. Perone Roberto Silveira Thomas S. Paula ODL UQCV 59 2 0 09 Jul 2021
Relational graph convolutional networks for predicting blood-brain barrier penetration of drug molecules Yan Ding Xiaoqian Jiang Yejin Kim GNN 46 19 0 04 Jul 2021
Tensor Normal Training for Deep Learning Models Yi Ren Shiqian Ma 105 28 0 05 Jun 2021
A Generalizable Approach to Learning Optimizers Diogo Almeida Clemens Winter Jie Tang Wojciech Zaremba AI4CE 93 29 0 02 Jun 2021
Surrogate Model Based Hyperparameter Tuning for Deep Learning with SPOT Thomas Bartz-Beielstein Frederik Rehbach Amrita Sen Martin Zaefferer 73 0 0 30 May 2021
On the Distributional Properties of Adaptive Gradients Z. Zhiyi Liu Ziyin 53 4 0 15 May 2021
Noether's Learning Dynamics: Role of Symmetry Breaking in Neural Networks Hidenori Tanaka D. Kunin 113 31 0 06 May 2021
Model Selection's Disparate Impact in Real-World Deep Learning Applications Jessica Zosa Forde A. Feder Cooper Kweku Kwegyir-Aggrey Chris De Sa Michael Littman 51 23 0 01 Apr 2021
Bump Hunting in Latent Space Blaž Bortolato B. Dillon J. Kamenik Aleks Smolkovič DRL 84 44 0 11 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 1.1K 30,096 0 26 Feb 2021
Revisiting the Role of Euler Numerical Integration on Acceleration and Stability in Convex Optimization Peiyuan Zhang Antonio Orvieto Hadi Daneshmand Thomas Hofmann Roy S. Smith 52 9 0 23 Feb 2021
Complex Momentum for Optimization in Games Jonathan Lorraine David Acuna Paul Vicol David Duvenaud 77 9 0 16 Feb 2021
Kronecker-factored Quasi-Newton Methods for Deep Learning Yi Ren Achraf Bahamou Shiqian Ma ODL 29 2 0 12 Feb 2021
A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes Zachary Nado Justin M. Gilmer Christopher J. Shallue Rohan Anil George E. Dahl ODL 104 27 0 12 Feb 2021
Hyperparameter Optimization Is Deceiving Us, and How to Stop It A. Feder Cooper Yucheng Lu Jessica Zosa Forde Christopher De Sa 45 32 0 05 Feb 2021
Adaptivity without Compromise: A Momentumized, Adaptive, Dual Averaged Gradient Method for Stochastic Optimization Aaron Defazio Samy Jelassi ODL 81 70 0 26 Jan 2021
Gravity Optimizer: a Kinematic Approach on Optimization in Deep Learning Dariush Bahrami Sadegh Pouriyan Zadeh ODL 44 5 0 22 Jan 2021
Classification and reconstruction of optical quantum states with deep neural networks Shahnawaz Ahmed C. Muñoz Franco Nori A. F. Kockum 88 64 0 03 Dec 2020
A Random Matrix Theory Approach to Damping in Deep Learning Diego Granziol Nicholas P. Baskerville AI4CE ODL 99 2 0 15 Nov 2020
Self-Tuning Stochastic Optimization with Curvature-Aware Gradient Filtering Ricky T. Q. Chen Dami Choi Lukas Balles David Duvenaud Philipp Hennig ODL 88 6 0 09 Nov 2020
Reverse engineering learned optimizers reveals known and novel mechanisms Niru Maheswaranathan David Sussillo Luke Metz Ruoxi Sun Jascha Narain Sohl-Dickstein 99 22 0 04 Nov 2020
Adaptive Gradient Quantization for Data-Parallel SGD Fartash Faghri Iman Tabrizian I. Markov Dan Alistarh Daniel M. Roy Ali Ramezani-Kebrya MQ 63 84 0 23 Oct 2020
Dual Averaging is Surprisingly Effective for Deep Learning Optimization Samy Jelassi Aaron Defazio 61 5 0 20 Oct 2020
How much progress have we made in neural network training? A New Evaluation Protocol for Benchmarking Optimizers Yuanhao Xiong Xuanqing Liu Li-Cheng Lan Yang You Si Si Cho-Jui Hsieh OOD 97 1 0 19 Oct 2020
Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves Luke Metz Niru Maheswaranathan C. Freeman Ben Poole Jascha Narain Sohl-Dickstein 156 61 0 23 Sep 2020
Review: Deep Learning in Electron Microscopy Jeffrey M. Ede 197 80 0 17 Sep 2020
ClipUp: A Simple and Powerful Optimizer for Distribution-based Policy Evolution N. E. Toklu Paweł Liskowski R. Srivastava 52 11 0 05 Aug 2020
Doubly infinite residual neural networks: a diffusion process approach Stefano Peluchetti Stefano Favaro 39 2 0 07 Jul 2020
Descending through a Crowded Valley - Benchmarking Deep Learning Optimizers Robin M. Schmidt Frank Schneider Philipp Hennig ODL 225 169 0 03 Jul 2020
AdaSGD: Bridging the gap between SGD and Adam Jiaxuan Wang Jenna Wiens 77 10 0 30 Jun 2020
Is SGD a Bayesian sampler? Well, almost Chris Mingard Guillermo Valle Pérez Joar Skalse A. Louis BDL 81 53 0 26 Jun 2020
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift Zachary Nado Shreyas Padhy D. Sculley Alexander DÁmour Balaji Lakshminarayanan Jasper Snoek OOD AI4TS 132 251 0 19 Jun 2020
Learning to Evaluate Perception Models Using Planner-Centric Metrics Jonah Philion Amlan Kar Sanja Fidler 63 69 0 19 Apr 2020
Iterative Averaging in the Quest for Best Test Error Diego Granziol Xingchen Wan Samuel Albanie Stephen J. Roberts 74 3 0 02 Mar 2020
Using a thousand optimization tasks to learn hyperparameter search strategies Luke Metz Niru Maheswaranathan Ruoxi Sun C. Freeman Ben Poole Jascha Narain Sohl-Dickstein 119 46 0 27 Feb 2020
Disentangling Adaptive Gradient Methods from Learning Rates Naman Agarwal Rohan Anil Elad Hazan Tomer Koren Cyril Zhang 109 38 0 26 Feb 2020