Variance Reduction on General Adaptive Stochastic Mirror Descent

26 December 2020

Papers citing "Variance Reduction on General Adaptive Stochastic Mirror Descent"

28 / 28 papers shown

Title
SVRG Meets AdaGrad: Painless Variance Reduction Benjamin Dubois-Taine Sharan Vaswani Reza Babanezhad Mark Schmidt Simon Lacoste-Julien 29 18 0 18 Feb 2021
Adam $^+$ : A Stochastic Method with Adaptive Variance Reduction Mingrui Liu Wei Zhang Francesco Orabona Tianbao Yang 34 28 0 24 Nov 2020
AdaBelief Optimizer: Adapting Stepsizes by the Belief in Observed Gradients Juntang Zhuang Tommy M. Tang Yifan Ding S. Tatikonda Nicha Dvornek X. Papademetris James S. Duncan ODL 83 505 0 15 Oct 2020
PAGE: A Simple and Optimal Probabilistic Gradient Estimator for Nonconvex Optimization Zhize Li Hongyan Bao Xiangliang Zhang Peter Richtárik ODL 49 128 0 25 Aug 2020
AdaX: Adaptive Gradient Descent with Exponential Long Term Memory Wenjie Li Zhaoyang Zhang Xinjiang Wang Ping Luo ODL 35 28 0 21 Apr 2020
On the Variance of the Adaptive Learning Rate and Beyond Liyuan Liu Haoming Jiang Pengcheng He Weizhu Chen Xiaodong Liu Jianfeng Gao Jiawei Han ODL 152 1,894 0 08 Aug 2019
A Hybrid Stochastic Optimization Framework for Stochastic Composite Nonconvex Optimization Quoc Tran-Dinh Nhan H. Pham T. Dzung Lam M. Nguyen 38 50 0 08 Jul 2019
Stabilized SVRG: Simple Variance Reduction for Nonconvex Optimization Rong Ge Zhize Li Weiyao Wang Xiang Wang 36 34 0 01 May 2019
On the Convergence of Adam and Beyond Sashank J. Reddi Satyen Kale Surinder Kumar 52 2,482 0 19 Apr 2019
On the Adaptivity of Stochastic Gradient-Based Optimization Lihua Lei Michael I. Jordan ODL 50 22 0 09 Apr 2019
Adaptive Gradient Methods with Dynamic Bound of Learning Rate Liangchen Luo Yuanhao Xiong Yan Liu Xu Sun ODL 46 600 0 26 Feb 2019
ProxSARAH: An Efficient Algorithmic Framework for Stochastic Composite Nonconvex Optimization Nhan H. Pham Lam M. Nguyen Dzung Phan Quoc Tran-Dinh 33 140 0 15 Feb 2019
On the Convergence of Adaptive Gradient Methods for Nonconvex Optimization Dongruo Zhou Yiqi Tang Yuan Cao Ziyan Yang Quanquan Gu 40 150 0 16 Aug 2018
On the Convergence of A Class of Adam-Type Algorithms for Non-Convex Optimization Xiangyi Chen Sijia Liu Ruoyu Sun Mingyi Hong 46 322 0 08 Aug 2018
SPIDER: Near-Optimal Non-Convex Optimization via Stochastic Path Integrated Differential Estimator Cong Fang C. J. Li Zhouchen Lin Tong Zhang 81 572 0 04 Jul 2018
Stochastic Nested Variance Reduction for Nonconvex Optimization Dongruo Zhou Pan Xu Quanquan Gu 49 147 0 20 Jun 2018
Nostalgic Adam: Weighting more of the past gradients when designing the adaptive learning rate Haiwen Huang Changzhang Wang Bin Dong ODL 22 59 0 19 May 2018
A Simple Proximal Stochastic Gradient Method for Nonsmooth Nonconvex Optimization Zhize Li Jian Li 56 116 0 13 Feb 2018
Natasha 2: Faster Non-Convex Optimization Than SGD Zeyuan Allen-Zhu ODL 64 245 0 29 Aug 2017
Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour Priya Goyal Piotr Dollár Ross B. Girshick P. Noordhuis Lukasz Wesolowski Aapo Kyrola Andrew Tulloch Yangqing Jia Kaiming He 3DH 91 3,666 0 08 Jun 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 48 1,023 0 23 May 2017
Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter Zeyuan Allen-Zhu 57 80 0 02 Feb 2017
Linear Convergence of Gradient and Proximal-Gradient Methods Under the Polyak-Łojasiewicz Condition Hamed Karimi J. Nutini Mark Schmidt 221 1,208 0 16 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 229 8,030 0 13 Aug 2016
Stochastic Variance Reduction for Nonconvex Optimization Sashank J. Reddi Ahmed S. Hefny S. Sra Barnabás Póczós Alex Smola 80 598 0 19 Mar 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.4K 192,638 0 10 Dec 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 808 149,474 0 22 Dec 2014
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives Aaron Defazio Francis R. Bach Simon Lacoste-Julien ODL 105 1,817 0 01 Jul 2014