v1v2 (latest)

Sharpness-Aware Minimization Leads to Low-Rank Features

25 May 2023

Maksym Andriushchenko

ArXiv (abs)PDF HTML Github (28★)

Papers citing "Sharpness-Aware Minimization Leads to Low-Rank Features"

24 / 24 papers shown

Title
Loss Gradient Gaussian Width based Generalization and Optimization Guarantees A. Banerjee Qiaobo Li Yingxue Zhou 147 0 0 11 Jun 2024
Momentum-SAM: Sharpness Aware Minimization without Computational Overhead Marlon Becker Frederick Altrock Benjamin Risse 155 6 0 22 Jan 2024
Critical Influence of Overparameterization on Sharpness-aware Minimization Sungbin Shin Dongyeop Lee Maksym Andriushchenko Namhoon Lee AAML 154 2 0 29 Nov 2023
The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima Peter L. Bartlett Philip M. Long Olivier Bousquet 157 37 0 04 Oct 2022
An Empirical Study of Implicit Regularization in Deep Offline RL Çağlar Gülçehre Srivatsan Srinivasan Jakub Sygnowski Georg Ostrovski Mehrdad Farajtabar Matt Hoffman Razvan Pascanu Arnaud Doucet OffRL 79 17 0 05 Jul 2022
Towards Understanding Sharpness-Aware Minimization Maksym Andriushchenko Nicolas Flammarion AAML 103 142 0 13 Jun 2022
Hierarchical Text-Conditional Image Generation with CLIP Latents Aditya A. Ramesh Prafulla Dhariwal Alex Nichol Casey Chu Mark Chen VLM DiffM 425 6,921 0 13 Apr 2022
Implicit Regularization Towards Rank Minimization in ReLU Networks Nadav Timor Gal Vardi Ohad Shamir 83 51 0 30 Jan 2022
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers Andreas Steiner Alexander Kolesnikov Xiaohua Zhai Ross Wightman Jakob Uszkoreit Lucas Beyer ViT 131 637 0 18 Jun 2021
MLP-Mixer: An all-MLP Architecture for Vision Ilya O. Tolstikhin N. Houlsby Alexander Kolesnikov Lucas Beyer Xiaohua Zhai ... Andreas Steiner Daniel Keysers Jakob Uszkoreit Mario Lucic Alexey Dosovitskiy 446 2,694 0 04 May 2021
On Feature Decorrelation in Self-Supervised Learning Tianyu Hua Wenxiao Wang Zihui Xue Sucheng Ren Yue Wang Hang Zhao SSL OOD 189 196 0 02 May 2021
ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks Jungmin Kwon Jeongseop Kim Hyunseong Park I. Choi 117 290 0 23 Feb 2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent Samuel L. Smith Benoit Dherin David Barrett Soham De MLT 47 204 0 28 Jan 2021
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 260 4,076 0 20 Nov 2020
Regularizing Neural Networks via Adversarial Model Perturbation Yaowei Zheng Richong Zhang Yongyi Mao AAML 98 99 0 10 Oct 2020
Sharpness-Aware Minimization for Efficiently Improving Generalization Pierre Foret Ariel Kleiner H. Mobahi Behnam Neyshabur AAML 201 1,360 0 03 Oct 2020
Implicit Gradient Regularization David Barrett Benoit Dherin 91 152 0 23 Sep 2020
Prevalence of Neural Collapse during the terminal phase of deep learning training Vardan Papyan Xuemei Han D. Donoho 219 582 0 18 Aug 2020
Fantastic Generalization Measures and Where to Find Them Yiding Jiang Behnam Neyshabur H. Mobahi Dilip Krishnan Samy Bengio AI4CE 145 611 0 04 Dec 2019
Bad Global Minima Exist and SGD Can Reach Them Shengchao Liu Dimitris Papailiopoulos D. Achlioptas 73 80 0 06 Jun 2019
On Lazy Training in Differentiable Programming Lénaïc Chizat Edouard Oyallon Francis R. Bach 111 840 0 19 Dec 2018
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 356 4,638 0 10 Nov 2016
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 444 2,947 0 15 Sep 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 357 10,204 0 16 Mar 2016