Provable Super-Convergence with a Large Cyclical Learning Rate

22 February 2021

Papers citing "Provable Super-Convergence with a Large Cyclical Learning Rate"

22 / 22 papers shown

Title
Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability Jeremy M. Cohen Simran Kaur Yuanzhi Li J. Zico Kolter Ameet Talwalkar ODL 73 267 0 26 Feb 2021
The Two Regimes of Deep Network Training Guillaume Leclerc Aleksander Madry 59 45 0 24 Feb 2020
Neural Spectrum Alignment: Empirical Study Dmitry Kopitkov Vadim Indelman 49 14 0 19 Oct 2019
Hessian based analysis of SGD for Deep Nets: Dynamics and Generalization Xinyan Li Qilong Gu Yingxue Zhou Tiancong Chen A. Banerjee ODL 69 52 0 24 Jul 2019
Global Convergence of Policy Gradient Methods to (Almost) Locally Optimal Policies Kai Zhang Alec Koppel Haoqi Zhu Tamer Basar 63 189 0 19 Jun 2019
Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobian Samet Oymak Zalan Fabian Mingchen Li Mahdi Soltanolkotabi MLT 58 88 0 12 Jun 2019
Gradient Descent with Early Stopping is Provably Robust to Label Noise for Overparameterized Neural Networks Mingchen Li Mahdi Soltanolkotabi Samet Oymak NoLa 93 352 0 27 Mar 2019
Cyclical Annealing Schedule: A Simple Approach to Mitigating KL Vanishing Hao Fu Chunyuan Li Xiaodong Liu Jianfeng Gao Asli Celikyilmaz Lawrence Carin ODL 75 363 0 25 Mar 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 196 1,099 0 18 Feb 2019
Reconciling modern machine learning practice and the bias-variance trade-off M. Belkin Daniel J. Hsu Siyuan Ma Soumik Mandal 221 1,638 0 28 Dec 2018
Gradient Descent Happens in a Tiny Subspace Guy Gur-Ari Daniel A. Roberts Ethan Dyer 88 233 0 12 Dec 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 252 3,194 0 20 Jun 2018
Overfitting or perfect fitting? Risk bounds for classification and regression rules that interpolate M. Belkin Daniel J. Hsu P. Mitra AI4CE 135 258 0 13 Jun 2018
Escaping Saddles with Stochastic Gradients Hadi Daneshmand Jonas Köhler Aurelien Lucchi Thomas Hofmann 53 162 0 15 Mar 2018
Averaging Weights Leads to Wider Optima and Better Generalization Pavel Izmailov Dmitrii Podoprikhin T. Garipov Dmitry Vetrov A. Wilson FedML MoMe 112 1,658 0 14 Mar 2018
Theory of Deep Learning III: explaining the non-overfitting puzzle T. Poggio Kenji Kawaguchi Q. Liao Brando Miranda Lorenzo Rosasco Xavier Boix Jack Hidary H. Mhaskar ODL 53 128 0 30 Dec 2017
Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates L. Smith Nicholay Topin AI4CE 82 522 0 23 Aug 2017
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks Levent Sagun Utku Evci V. U. Güney Yann N. Dauphin Léon Bottou 54 418 0 14 Jun 2017
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark Schmidt 262 1,218 0 16 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 288 8,091 0 13 Aug 2016
Cyclical Learning Rates for Training Neural Networks L. Smith ODL 190 2,517 0 03 Jun 2015
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.5K 100,330 0 04 Sep 2014