v1v2v3v4 (latest)

Demon: Improved Neural Network Training with Momentum Decay

11 October 2019

John Chen

Cameron R. Wolfe

Zhaoqi Li

Anastasios Kyrillidis

ODL

ArXiv (abs)PDF HTML

Papers citing "Demon: Improved Neural Network Training with Momentum Decay"

50 / 53 papers shown

Title
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 904 42,463 0 28 May 2020
Decision-Making with Auto-Encoding Variational Bayes Romain Lopez Pierre Boyeau Nir Yosef Michael I. Jordan Jeffrey Regier BDL 550 10,591 0 17 Feb 2020
Optimizer Benchmarking Needs to Account for Hyperparameter Tuning Prabhu Teja Sivaprasad Florian Mai Thijs Vogels Martin Jaggi François Fleuret 75 12 0 25 Oct 2019
On Empirical Comparisons of Optimizers for Deep Learning Dami Choi Christopher J. Shallue Zachary Nado Jaehoon Lee Chris J. Maddison George E. Dahl 109 259 0 11 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 267 7,554 0 02 Oct 2019
Generative Modeling by Estimating Gradients of the Data Distribution Yang Song Stefano Ermon SyDa DiffM 258 3,961 0 12 Jul 2019
Reducing the variance in online optimization by transporting past gradients Sébastien M. R. Arnold Pierre-Antoine Manzagol Reza Babanezhad Ioannis Mitliagkas Nicolas Le Roux 81 28 0 08 Jun 2019
Momentum-Based Variance Reduction in Non-Convex SGD Ashok Cutkosky Francesco Orabona ODL 96 410 0 24 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 109 2,506 0 19 Apr 2019
A Sufficient Condition for Convergences of Adam and RMSProp Fangyu Zou Li Shen Zequn Jie Weizhong Zhang Wei Liu 71 372 0 23 Nov 2018
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 154 130 0 16 Oct 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Negative Momentum for Improved Game Dynamics Gauthier Gidel Reyhane Askari Hemmat Mohammad Pezeshki Rémi Le Priol Gabriel Huang Simon Lacoste-Julien Ioannis Mitliagkas AI4CE 77 181 0 12 Jul 2018
Closing the Generalization Gap of Adaptive Gradient Methods in Training Deep Neural Networks Jinghui Chen Dongruo Zhou Yiqi Tang Ziyan Yang Yuan Cao Quanquan Gu ODL 101 193 0 18 Jun 2018
Neural Network Acceptability Judgments Alex Warstadt Amanpreet Singh Samuel R. Bowman 255 1,413 0 31 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,201 0 20 Apr 2018
Aggregated Momentum: Stability Through Passive Damping James Lucas Shengyang Sun R. Zemel Roger C. Grosse 70 68 0 01 Apr 2018
A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay L. Smith 286 1,035 0 26 Mar 2018
On the insufficiency of existing momentum schemes for Stochastic Optimization Rahul Kidambi Praneeth Netrapalli Prateek Jain Sham Kakade ODL 90 120 0 15 Mar 2018
ADINE: An Adaptive Momentum Method for Stochastic Gradient Descent Vishwak Srinivasan Adepu Ravi Sankar V. Balasubramanian ODL 37 16 0 20 Dec 2017
Decoupled Weight Decay Regularization I. Loshchilov Frank Hutter OffRL 154 2,158 0 14 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 117 996 0 01 Nov 2017
Dynamic Routing Between Capsules S. Sabour Nicholas Frosst Geoffrey E. Hinton 195 4,613 0 26 Oct 2017
Squeeze-and-Excitation Networks Jie Hu Li Shen Samuel Albanie Gang Sun Enhua Wu 427 26,605 0 05 Sep 2017
YellowFin and the Art of Momentum Tuning Jian Zhang Ioannis Mitliagkas ODL 83 108 0 12 Jun 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 91 1,032 0 23 May 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 174 3,292 0 08 May 2017
Residual Attention Network for Image Classification Fei Wang Mengqing Jiang Chao Qian Shuo Yang Cheng Li Honggang Zhang Xiaogang Wang Xiaoou Tang 117 3,314 0 23 Apr 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 526 4,497 0 18 Apr 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,900 0 17 Apr 2017
Wasserstein GAN Martín Arjovsky Soumith Chintala Léon Bottou GAN 183 4,829 0 26 Jan 2017
Feature Pyramid Networks for Object Detection Nayeon Lee Piotr Dollár Ross B. Girshick Kaiming He Bharath Hariharan Serge J. Belongie ObjD 491 22,158 0 09 Dec 2016
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 524 10,358 0 16 Nov 2016
An overview of gradient descent optimization algorithms Sebastian Ruder ODL 213 6,206 0 15 Sep 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 849 36,910 0 25 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 350 8,179 0 13 Aug 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 316 8,177 0 16 Jun 2016
Fast Zero-Shot Image Tagging Yang Zhang Boqing Gong M. Shah VLM 3DV 65 142 0 31 May 2016
TensorFlow: A system for large-scale machine learning Martín Abadi P. Barham Jianmin Chen Zhiwen Chen Andy Davis ... Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zhang GNN AI4CE 435 18,361 0 27 May 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 362 8,005 0 23 May 2016
Revisiting Distributed Synchronous SGD Jianmin Chen Xinghao Pan R. Monga Samy Bengio Rafal Jozefowicz 89 801 0 04 Apr 2016
A Variational Perspective on Accelerated Methods in Optimization Andre Wibisono Ashia Wilson Michael I. Jordan 95 575 0 14 Mar 2016
On the Influence of Momentum Acceleration on Online Learning Kun Yuan Bicheng Ying Ali H. Sayed 76 58 0 14 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.3K 194,641 0 10 Dec 2015
Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Alec Radford Luke Metz Soumith Chintala GAN OOD 309 14,032 0 19 Nov 2015
Very Deep Multilingual Convolutional Neural Networks for LVCSR Tom Sercu Christian Puhrsch Brian Kingsbury Yann LeCun 79 224 0 29 Sep 2015
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren Kaiming He Ross B. Girshick Jian Sun AIMat ObjD 537 62,477 0 04 Jun 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,433 0 22 Dec 2014
Conditional Generative Adversarial Nets M. Berk Mirza Simon Osindero GAN SyDa AI4CE 269 10,438 0 06 Nov 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.7K 100,575 0 04 Sep 2014