How to decay your learning rate

23 March 2021

Papers citing "How to decay your learning rate"

22 / 22 papers shown

Title
Reconciling Modern Deep Learning with Traditional Optimization Analyses: The Intrinsic Learning Rate Zhiyuan Li Kaifeng Lyu Sanjeev Arora 88 75 0 06 Oct 2020
Sharpness-Aware Minimization for Efficiently Improving Generalization Pierre Foret Ariel Kleiner H. Mobahi Behnam Neyshabur AAML 192 1,350 0 03 Oct 2020
MLRSNet: A Multi-label High Spatial Resolution Remote Sensing Dataset for Semantic Scene Understanding Xiaoman Qi P. Zhu Yuebin Wang Liqiang Zhang Junhuan Peng Mengfan Wu Jialong Chen Xudong Zhao Ning Zang P. Mathiopoulos 86 116 0 01 Oct 2020
On the training dynamics of deep networks with $L_2$ regularization Aitor Lewkowycz Guy Gur-Ari 90 53 0 15 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 774 42,055 0 28 May 2020
Learning Rate Annealing Can Provably Help Generalization, Even for Convex Problems Preetum Nakkiran MLT 52 21 0 15 May 2020
The large learning rate phase of deep learning: the catapult mechanism Aitor Lewkowycz Yasaman Bahri Ethan Dyer Jascha Narain Sohl-Dickstein Guy Gur-Ari ODL 191 241 0 04 Mar 2020
Disentangling Adaptive Gradient Methods from Learning Rates Naman Agarwal Rohan Anil Elad Hazan Tomer Koren Cyril Zhang 85 37 0 26 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 602 4,822 0 23 Jan 2020
An Exponential Learning Rate Schedule for Deep Learning Zhiyuan Li Sanjeev Arora 51 218 0 16 Oct 2019
Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks Yuanzhi Li Colin Wei Tengyu Ma 47 299 0 10 Jul 2019
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks Kaifeng Lyu Jian Li 84 335 0 13 Jun 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 139 18,134 0 28 May 2019
Measuring the Effects of Data Parallelism on Neural Network Training Christopher J. Shallue Jaehoon Lee J. Antognini J. Mamou J. Ketterling Yao Wang 82 409 0 08 Nov 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 94,891 0 11 Oct 2018
Fluctuation-dissipation relations for stochastic gradient descent Sho Yaida 86 74 0 28 Sep 2018
L4: Practical loss-based stepsize adaptation for deep learning Michal Rolínek Georg Martius ODL 107 64 0 14 Feb 2018
Meta-SGD: Learning to Learn Quickly for Few-Shot Learning Zhenguo Li Fengwei Zhou Fei Chen Hang Li 94 1,119 0 31 Jul 2017
L2 Regularization versus Batch and Weight Normalization Twan van Laarhoven 65 300 0 16 Jun 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017
Learned Optimizers that Scale and Generalize Olga Wichrowska Niru Maheswaranathan Matthew W. Hoffman Sergio Gomez Colmenarejo Misha Denil Nando de Freitas Jascha Narain Sohl-Dickstein AI4CE 56 283 0 14 Mar 2017
Gradient-based Hyperparameter Optimization through Reversible Learning D. Maclaurin David Duvenaud Ryan P. Adams DD 227 945 0 11 Feb 2015