v1v2v3v4v5 (latest)

On the Performance Analysis of Momentum Method: A Frequency Domain Perspective

29 November 2024

Papers citing "On the Performance Analysis of Momentum Method: A Frequency Domain Perspective"

30 / 30 papers shown

Title
The Marginal Value of Momentum for Small Learning Rate SGD Runzhe Wang Sadhika Malladi Tianhao Wang Kaifeng Lyu Zhiyuan Li ODL 75 9 0 27 Jul 2023
Tianshou: a Highly Modularized Deep Reinforcement Learning Library Jiayi Weng Huayu Chen Dong Yan Kaichao You Alexis Duburcq Minghao Zhang Yi Su Hang Su Jun Zhu NoLa OffRL 97 204 0 29 Jul 2021
On the Last Iterate Convergence of Momentum Methods Xiaoyun Li Mingrui Liu Francesco Orabona 47 12 0 13 Feb 2021
On the Generalization Benefit of Noise in Stochastic Gradient Descent Samuel L. Smith Erich Elsen Soham De MLT 57 100 0 26 Jun 2020
Convergence of a Stochastic Gradient Method with Momentum for Non-Smooth Non-Convex Optimization Vien V. Mai M. Johansson 77 56 0 13 Feb 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 562 42,639 0 03 Dec 2019
A Generic Acceleration Framework for Stochastic Composite Optimization A. Kulunchakov Julien Mairal 91 43 0 03 Jun 2019
Momentum-Based Variance Reduction in Non-Convex SGD Ashok Cutkosky Francesco Orabona ODL 96 410 0 24 May 2019
On the Linear Speedup Analysis of Communication Efficient Momentum SGD for Distributed Non-Convex Optimization Hao Yu Rong Jin Sen Yang FedML 100 387 0 09 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 109 2,506 0 19 Apr 2019
Adaptive Gradient Methods with Dynamic Bound of Learning Rate Liangchen Luo Yuanhao Xiong Yan Liu Xu Sun ODL 91 602 0 26 Feb 2019
Pay Less Attention with Lightweight and Dynamic Convolutions Felix Wu Angela Fan Alexei Baevski Yann N. Dauphin Michael Auli 89 610 0 29 Jan 2019
A Universally Optimal Multistage Accelerated Stochastic Gradient Method N. Aybat Alireza Fallah Mert Gurbuzbalaban Asuman Ozdaglar ODL 91 57 0 23 Jan 2019
Quasi-hyperbolic momentum and Adam for deep learning Jerry Ma Denis Yarats ODL 154 130 0 16 Oct 2018
A Unified Analysis of Stochastic Momentum Methods for Deep Learning Yan Yan Tianbao Yang Zhe Li Qihang Lin Yi Yang 45 120 0 30 Aug 2018
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization Xiangyi Chen Sijia Liu Ruoyu Sun Mingyi Hong 78 324 0 08 Aug 2018
A disciplined approach to neural network hyper-parameters: Part 1 -- learning rate, batch size, momentum, and weight decay L. Smith 286 1,035 0 26 Mar 2018
On the insufficiency of existing momentum schemes for Stochastic Optimization Rahul Kidambi Praneeth Netrapalli Prateek Jain Sham Kakade ODL 90 120 0 15 Mar 2018
Proximal Policy Optimization Algorithms John Schulman Filip Wolski Prafulla Dhariwal Alec Radford Oleg Klimov OffRL 565 19,296 0 20 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 174 3,290 0 08 May 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.2K 20,900 0 17 Apr 2017
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 825 36,892 0 25 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 350 8,179 0 13 Aug 2016
Sequence-to-Sequence Learning as Beam-Search Optimization Sam Wiseman Alexander M. Rush 141 594 0 09 Jun 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.3K 194,510 0 10 Dec 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 350 18,654 0 06 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,364 0 22 Dec 2014
Very Deep Convolutional Networks for Large-Scale Image Recognition Karen Simonyan Andrew Zisserman FAtt MDE 1.7K 100,529 0 04 Sep 2014
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 1.7K 39,615 0 01 Sep 2014