v1v2 (latest)

Train faster, generalize better: Stability of stochastic gradient descent

3 September 2015

Moritz Hardt

Benjamin Recht

Y. Singer

ArXiv (abs)PDF HTML

Papers citing "Train faster, generalize better: Stability of stochastic gradient descent"

50 / 679 papers shown

Title
Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspective Yuxin Dong Tieliang Gong Hao Chen Chen Li 75 3 0 02 May 2023
Oversampling Higher-Performing Minorities During Machine Learning Model Training Reduces Adverse Impact Slightly but Also Reduces Model Accuracy Louis Hickman J. Kuruzovich Vincent Ng Kofi Arhin Danielle Wilson 36 14 0 27 Apr 2023
Learning Trajectories are Generalization Indicators Jingwen Fu Zhizheng Zhang Dacheng Yin Yan Lu Nanning Zheng AI4CE 90 3 0 25 Apr 2023
AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs Shengrui Li Xueting Han Jing Bai AI4CE 45 12 0 19 Apr 2023
Differentially Private Stochastic Convex Optimization in (Non)-Euclidean Space Revisited Jinyan Su Changhong Zhao Di Wang 78 5 0 31 Mar 2023
Lipschitzness Effect of a Loss Function on Generalization Performance of Deep Neural Networks Trained by Adam and AdamW Optimizers M. Lashkari Amin Gheibi 112 3 0 29 Mar 2023
Lower Generalization Bounds for GD and SGD in Smooth Stochastic Convex Optimization Peiyuan Zhang Jiaye Teng J.N. Zhang 78 4 0 19 Mar 2023
Neural Frailty Machine: Beyond proportional hazard assumption in neural survival regressions Ruofan Wu Jiawei Qiao Mingzhe Wu Wen Yu M. Zheng Tengfei Liu Tianyi Zhang Weiqiang Wang 61 1 0 18 Mar 2023
Benign Overfitting for Two-layer ReLU Convolutional Neural Networks Yiwen Kou Zi-Yuan Chen Yuanzhou Chen Quanquan Gu MLT 94 17 0 07 Mar 2023
Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization Xingxuan Zhang Renzhe Xu Han Yu Hao Zou Peng Cui 83 42 0 03 Mar 2023
Learning to Generalize Provably in Learning to Optimize Junjie Yang Tianlong Chen Mingkang Zhu Fengxiang He Dacheng Tao Yitao Liang Zhangyang Wang 86 7 0 22 Feb 2023
Stability-based Generalization Analysis for Mixtures of Pointwise and Pairwise Learning Jiahuan Wang Jun Chen Hao Chen Bin Gu Weifu Li Xinwei Tang MLT 70 2 0 20 Feb 2023
On the Stability and Generalization of Triplet Learning Jun Chen Hao Chen Xue Jiang Bin Gu Weifu Li Tieliang Gong Feng Zheng 59 4 0 20 Feb 2023
Why Is Public Pretraining Necessary for Private Model Training? Arun Ganesh Mahdi Haghifam Milad Nasr Sewoong Oh Thomas Steinke Om Thakkar Abhradeep Thakurta Lun Wang 70 39 0 19 Feb 2023
Generalization and Stability of Interpolating Neural Networks with Minimal Width Hossein Taheri Christos Thrampoulidis 105 16 0 18 Feb 2023
Cyclic and Randomized Stepsizes Invoke Heavier Tails in SGD than Constant Stepsize Mert Gurbuzbalaban Yuanhan Hu Umut Simsekli Lingjiong Zhu LRM 95 1 0 10 Feb 2023
Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion Haotian Ju Dongyue Li Aneesh Sharma Hongyang R. Zhang 70 41 0 09 Feb 2023
U-Clip: On-Average Unbiased Stochastic Gradient Clipping Bryn Elesedy Marcus Hutter 64 1 0 06 Feb 2023
Efficient Gradient Approximation Method for Constrained Bilevel Optimization Siyuan Xu Minghui Zhu 78 20 0 03 Feb 2023
Bagging Provides Assumption-free Stability Jake A. Soloff Rina Foygel Barber Rebecca Willett 69 11 0 30 Jan 2023
Implicit Regularization for Group Sparsity Jiangyuan Li THANH VAN NGUYEN Chinmay Hegde Raymond K. W. Wong 96 9 0 29 Jan 2023
On the Lipschitz Constant of Deep Networks and Double Descent Matteo Gamba Hossein Azizpour Mårten Björkman 110 7 0 28 Jan 2023
Algorithmic Stability of Heavy-Tailed SGD with General Loss Functions Anant Raj Lingjiong Zhu Mert Gurbuzbalaban Umut Simsekli 86 16 0 27 Jan 2023
Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensing Jikai Jin Zhiyuan Li Kaifeng Lyu S. Du Jason D. Lee MLT 118 37 0 27 Jan 2023
A Stability Analysis of Fine-Tuning a Pre-Trained Model Z. Fu Anthony Man-Cho So Nigel Collier 74 3 0 24 Jan 2023
Stretched and measured neural predictions of complex network dynamics V. Vasiliauskaite Nino Antulov-Fantulin 82 1 0 12 Jan 2023
Sharper Analysis for Minibatch Stochastic Proximal Point Methods: Stability, Smoothness, and Deviation Xiao-Tong Yuan P. Li 93 2 0 09 Jan 2023
Resampling Sensitivity of High-Dimensional PCA Haoyu Wang 62 0 0 30 Dec 2022
Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimization Mahdi Haghifam Borja Rodríguez Gálvez Ragnar Thobaben Mikael Skoglund Daniel M. Roy Gintare Karolina Dziugaite 93 19 0 27 Dec 2022
Iterative regularization in classification via hinge loss diagonal descent Vassilis Apidopoulos T. Poggio Lorenzo Rosasco S. Villa 67 2 0 24 Dec 2022
On the Overlooked Structure of Stochastic Gradients Zeke Xie Qian-Yuan Tang Mingming Sun P. Li 98 6 0 05 Dec 2022
On the Effectiveness of Parameter-Efficient Fine-Tuning Z. Fu Haoran Yang Anthony Man-Cho So Wai Lam Lidong Bing Nigel Collier 80 162 0 28 Nov 2022
SIFU: Sequential Informed Federated Unlearning for Efficient and Provable Client Unlearning in Federated Optimization Yann Fraboni Martin Van Waerebeke Kevin Scaman Richard Vidal Laetitia Kameni Marco Lorenzi FedML MU 116 14 0 21 Nov 2022
Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal States Ziqiao Wang Yongyi Mao 111 12 0 19 Nov 2022
SketchySGD: Reliable Stochastic Optimization via Randomized Curvature Estimates Zachary Frangella Pratik Rathore Shipu Zhao Madeleine Udell 55 6 0 16 Nov 2022
On the Algorithmic Stability and Generalization of Adaptive Optimization Methods Han Nguyen Hai Pham Sashank J. Reddi Barnabas Poczos ODL AI4CE 101 2 0 08 Nov 2022
Do highly over-parameterized neural networks generalize since bad solutions are rare? Julius Martinetz T. Martinetz 91 1 0 07 Nov 2022
Distributed DP-Helmet: Scalable Differentially Private Non-interactive Averaging of Single Layers Moritz Kirschte Sebastian Meiser Saman Ardalan Esfandiar Mohammadi FedML 76 0 0 03 Nov 2022
Optimal Algorithms for Stochastic Complementary Composite Minimization Alexandre d’Aspremont Cristóbal Guzmán Clément Lezane 64 3 0 03 Nov 2022
Variance reduced Shapley value estimation for trustworthy data valuation Mengmeng Wu R. Jia Changle Lin Wei Huang Xiangyu Chang TDI 127 15 0 30 Oct 2022
Chaos Theory and Adversarial Robustness Jonathan S. Kent AAML 74 0 0 20 Oct 2022
Stochastic Differentially Private and Fair Learning Andrew Lowy Devansh Gupta Meisam Razaviyayn FaML FedML 61 14 0 17 Oct 2022
FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation Ming Hu Peiheng Zhou Zhihao Yue Zhiwei Ling Yihao Huang Anran Li Yang Liu Xiang Lian Mingsong Chen FedML 79 14 0 15 Oct 2022
On the Importance of Gradient Norm in PAC-Bayesian Bounds Itai Gat Yossi Adi Alex Schwing Tamir Hazan BDL 97 6 0 12 Oct 2022
On Stability and Generalization of Bilevel Optimization Problem Meng Ding Ming Lei Yunwen Lei Di Wang Jinhui Xu 103 1 0 03 Oct 2022
Stability Analysis and Generalization Bounds of Adversarial Training Jiancong Xiao Yanbo Fan Ruoyu Sun Jue Wang Zhimin Luo AAML 85 31 0 03 Oct 2022
Adaptive Smoothness-weighted Adversarial Training for Multiple Perturbations with Its Stability Analysis Jiancong Xiao Zeyu Qin Yanbo Fan Baoyuan Wu Jue Wang Zhimin Luo AAML 131 7 0 02 Oct 2022
Neural Networks Efficiently Learn Low-Dimensional Representations with SGD Alireza Mousavi-Hosseini Sejun Park M. Girotti Ioannis Mitliagkas Murat A. Erdogdu MLT 379 50 0 29 Sep 2022
Exploring the Algorithm-Dependent Generalization of AUPRC Optimization with List Stability Peisong Wen Qianqian Xu Zhiyong Yang Yuan He Qingming Huang 138 10 0 27 Sep 2022
On the Stability Analysis of Open Federated Learning Systems Youbang Sun H. Fernando Tianyi Chen Shahin Shahrampour FedML 79 1 0 25 Sep 2022