Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization

4 October 2020

Papers citing "Quickly Finding a Benign Region via Heavy Ball Momentum in Non-Convex Optimization"

49 / 49 papers shown

Title
Escaping Saddle Points Faster with Stochastic Momentum Jun-Kun Wang Chi-Heng Lin Jacob D. Abernethy ODL 42 22 0 05 Jun 2021
Global Convergence of Second-order Dynamics in Two-layer Neural Networks Walid Krichene Kenneth F. Caluya A. Halder MLT 14 5 0 14 Jul 2020
Robust Recovery via Implicit Bias of Discrepant Learning Rates for Double Over-parameterization Chong You Zhihui Zhu Qing Qu Yi-An Ma 12 42 0 16 Jun 2020
Almost sure convergence rates for Stochastic Gradient Descent and Stochastic Heavy Ball Othmane Sebbouh Robert Mansel Gower Aaron Defazio 31 22 0 14 Jun 2020
Complex Dynamics in Simple Neural Networks: Understanding Gradient Flow in Phase Retrieval Stefano Sarao Mannelli Giulio Biroli C. Cammarota Florent Krzakala Pierfrancesco Urbani Lenka Zdeborová 20 28 0 12 Jun 2020
The Effects of Mild Over-parameterization on the Optimization Landscape of Shallow ReLU Neural Networks Itay Safran Gilad Yehudai Ohad Shamir 103 34 0 01 Jun 2020
Understanding the Role of Momentum in Stochastic Gradient Methods Igor Gitman Hunter Lang Pengchuan Zhang Lin Xiao 38 94 0 30 Oct 2019
Near-Optimal Methods for Minimizing Star-Convex Functions and Beyond Oliver Hinder Aaron Sidford N. Sohoni 31 71 0 27 Jun 2019
Generalized Momentum-Based Methods: A Hamiltonian Perspective Jelena Diakonikolas Michael I. Jordan 27 57 0 02 Jun 2019
Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks Gauthier Gidel Francis R. Bach Simon Lacoste-Julien AI4CE 40 153 0 30 Apr 2019
On Nonconvex Optimization for Machine Learning: Gradients, Stochasticity, and Saddle Points Chi Jin Praneeth Netrapalli Rong Ge Sham Kakade Michael I. Jordan 65 61 0 13 Feb 2019
Sharp Analysis for Nonconvex SGD Escaping from Saddle Points Cong Fang Zhouchen Lin Tong Zhang 50 104 0 01 Feb 2019
Escaping Saddle Points with Adaptive Gradient Methods Matthew Staib Sashank J. Reddi Satyen Kale Sanjiv Kumar S. Sra ODL 37 74 0 26 Jan 2019
Accelerated Linear Convergence of Stochastic Momentum Methods in Wasserstein Distances Bugra Can Mert Gurbuzbalaban Lingjiong Zhu 43 44 0 22 Jan 2019
Non-ergodic Convergence Analysis of Heavy-Ball Algorithms Tao Sun Penghang Yin Dongsheng Li Chun Huang Lei Guan Hao Jiang 17 46 0 05 Nov 2018
Accelerated Gossip via Stochastic Heavy Ball Method Nicolas Loizou Peter Richtárik 17 27 0 23 Sep 2018
Gradient Descent with Random Initialization: Fast Global Convergence for Nonconvex Phase Retrieval Yuxin Chen Yuejie Chi Jianqing Fan Cong Ma 39 235 0 21 Mar 2018
Escaping Saddles with Stochastic Gradients Hadi Daneshmand Jonas Köhler Aurelien Lucchi Thomas Hofmann 47 162 0 15 Mar 2018
On the insufficiency of existing momentum schemes for Stochastic Optimization Rahul Kidambi Praneeth Netrapalli Prateek Jain Sham Kakade ODL 60 118 0 15 Mar 2018
Nonconvex Matrix Factorization from Rank-One Measurements Yuanxin Li Cong Ma Yuxin Chen Yuejie Chi 39 51 0 17 Feb 2018
An Alternative View: When Does SGD Escape Local Minima? Robert D. Kleinberg Yuanzhi Li Yang Yuan MLT 41 316 0 17 Feb 2018
$Optimization-based AMP for Phase Retrieval: The Impact of Initialization and $\ell_2$-regularization$ Optimization-based AMP for Phase Retrieval: The Impact of Initialization and $\ell_2$ -regularization Junjie Ma Ji Xu A. Maleki 63 53 0 03 Jan 2018
Momentum and Stochastic Momentum for Stochastic Gradient, Newton, Proximal Point and Subspace Descent Methods Nicolas Loizou Peter Richtárik 53 200 0 27 Dec 2017
Convolutional Phase Retrieval via Gradient Descent Qing Qu Yuqian Zhang Yonina C. Eldar John N. Wright 68 29 0 03 Dec 2017
Implicit Regularization in Nonconvex Statistical Estimation: Gradient Descent Converges Linearly for Phase Retrieval, Matrix Completion, and Blind Deconvolution Cong Ma Kaizheng Wang Yuejie Chi Yuxin Chen 42 240 0 28 Nov 2017
Accelerated Gradient Descent Escapes Saddle Points Faster than Gradient Descent Chi Jin Praneeth Netrapalli Michael I. Jordan ODL 51 261 0 28 Nov 2017
Neon2: Finding Local Minima via First-Order Oracles Zeyuan Allen-Zhu Yuanzhi Li 49 130 0 17 Nov 2017
First-order Stochastic Algorithms for Escaping From Saddle Points in Almost Linear Time Yi Tian Xu Rong Jin Tianbao Yang ODL 39 116 0 03 Nov 2017
A Generic Approach for Escaping Saddle points Sashank J. Reddi Manzil Zaheer S. Sra Barnabás Póczós Francis R. Bach Ruslan Salakhutdinov Alex Smola 79 83 0 05 Sep 2017
Gradient Descent Can Take Exponential Time to Escape Saddle Points S. Du Chi Jin Jason D. Lee Michael I. Jordan Barnabás Póczós Aarti Singh 37 244 0 29 May 2017
Convergence Analysis of Two-layer Neural Networks with ReLU Activation Yuanzhi Li Yang Yuan MLT 62 649 0 28 May 2017
Implicit Regularization in Matrix Factorization Suriya Gunasekar Blake E. Woodworth Srinadh Bhojanapalli Behnam Neyshabur Nathan Srebro 52 490 0 25 May 2017
Train longer, generalize better: closing the generalization gap in large batch training of neural networks Elad Hoffer Itay Hubara Daniel Soudry ODL 134 798 0 24 May 2017
The Marginal Value of Adaptive Gradient Methods in Machine Learning Ashia Wilson Rebecca Roelofs Mitchell Stern Nathan Srebro Benjamin Recht ODL 41 1,023 0 23 May 2017
Solving (most) of a set of quadratic equalities: Composite optimization for robust phase retrieval John C. Duchi Feng Ruan 26 165 0 05 May 2017
How to Escape Saddle Points Efficiently Chi Jin Rong Ge Praneeth Netrapalli Sham Kakade Michael I. Jordan ODL 128 834 0 02 Mar 2017
The Power of Normalization: Faster Evasion of Saddle Points Kfir Y. Levy 44 108 0 15 Nov 2016
Stochastic Heavy Ball S. Gadat Fabien Panloup Sofiane Saadane 78 102 0 14 Sep 2016
Solving Systems of Random Quadratic Equations via Truncated Amplitude Flow G. Wang G. Giannakis Yonina C. Eldar 28 362 0 26 May 2016
Unified Convergence Analysis of Stochastic Momentum Methods for Convex and Non-convex Optimization Tianbao Yang Qihang Lin Zhe Li 41 121 0 12 Apr 2016
A Geometric Analysis of Phase Retrieval Ju Sun Qing Qu John N. Wright 43 524 0 22 Feb 2016
The local convexity of solving systems of quadratic equations Christopher D. White Sujay Sanghavi Rachel A. Ward 31 72 0 25 Jun 2015
A Convergent Gradient Descent Algorithm for Rank Minimization and Semidefinite Programming from Random Linear Measurements Qinqing Zheng John D. Lafferty 36 186 0 19 Jun 2015
Optimal Rates of Convergence for Noisy Sparse Phase Retrieval via Thresholded Wirtinger Flow T. Tony Cai Xiaodong Li Zongming Ma 50 232 0 10 Jun 2015
Solving Random Quadratic Systems of Equations Is Nearly as Easy as Solving Linear Systems Yuxin Chen Emmanuel J. Candes 43 589 0 19 May 2015
From Averaging to Acceleration, There is Only a Step-size Nicolas Flammarion Francis R. Bach 67 138 0 07 Apr 2015
Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decomposition Rong Ge Furong Huang Chi Jin Yang Yuan 107 1,056 0 06 Mar 2015
Phase Retrieval via Wirtinger Flow: Theory and Algorithms Emmanuel Candes Xiaodong Li Mahdi Soltanolkotabi 101 1,282 0 03 Jul 2014
Phase Retrieval using Alternating Minimization Praneeth Netrapalli Prateek Jain Sujay Sanghavi 134 632 0 02 Jun 2013