v1v2v3 (latest)

Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning

23 May 2023

ArXiv (abs)PDF HTML

Papers citing "Layer-wise Adaptive Step-Sizes for Stochastic First-Order Methods for Deep Learning"

15 / 15 papers shown

Title
Stochastic Polyak Stepsize with a Moving Target Robert Mansel Gower Aaron Defazio Michael G. Rabbat 61 17 0 22 Jun 2021
Tensor Normal Training for Deep Learning Models Yi Ren D. Goldfarb 69 28 0 05 Jun 2021
ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning Z. Yao A. Gholami Sheng Shen Mustafa Mustafa Kurt Keutzer Michael W. Mahoney ODL 108 285 0 01 Jun 2020
Stochastic Polyak Step-size for SGD: An Adaptive Learning Rate for Fast Convergence Nicolas Loizou Sharan Vaswani I. Laradji Simon Lacoste-Julien 69 187 0 24 Feb 2020
On Empirical Comparisons of Optimizers for Deep Learning Dami Choi Christopher J. Shallue Zachary Nado Jaehoon Lee Chris J. Maddison George E. Dahl 91 259 0 11 Oct 2019
Painless Stochastic Gradient: Interpolation, Line-Search, and Convergence Rates Sharan Vaswani Aaron Mishkin I. Laradji Mark Schmidt Gauthier Gidel Simon Lacoste-Julien ODL 86 210 0 24 May 2019
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density Behrooz Ghorbani Shankar Krishnan Ying Xiao ODL 83 326 0 29 Jan 2019
Shampoo: Preconditioned Stochastic Tensor Optimization Vineet Gupta Tomer Koren Y. Singer ODL 95 226 0 26 Feb 2018
FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling Jie Chen Tengfei Ma Cao Xiao GNN 149 1,517 0 30 Jan 2018
Improving Generalization Performance by Switching from Adam to SGD N. Keskar R. Socher ODL 103 524 0 20 Dec 2017
Practical Gauss-Newton Optimisation for Deep Learning Aleksandar Botev H. Ritter David Barber ODL 63 232 0 12 Jun 2017
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 793 36,881 0 25 Aug 2016
Optimizing Neural Networks with Kronecker-factored Approximate Curvature James Martens Roger C. Grosse ODL 104 1,023 0 19 Mar 2015
Probabilistic Line Searches for Stochastic Optimization Maren Mahsereci Philipp Hennig ODL 68 126 0 10 Feb 2015
ADADELTA: An Adaptive Learning Rate Method Matthew D. Zeiler ODL 165 6,632 0 22 Dec 2012

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.