v1v2 (latest)

Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances

25 May 2021

Papers citing "Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances"

38 / 38 papers shown

Title
Understanding Mode Connectivity via Parameter Space Symmetry B. Zhao Nima Dehmamy Robin Walters Rose Yu 225 8 0 29 May 2025
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries Chris Kolb T. Weber Bernd Bischl David Rügamer 339 1 0 04 Feb 2025
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion Binchi Zhang Zaiyi Zheng Zhengzhang Chen Wenlin Yao 200 1 0 01 Feb 2025
Learning Gaussian Multi-Index Models with Gradient Flow: Time Complexity and Directional Convergence Berfin Simsek Amire Bendjeddou Daniel Hsu 151 3 0 13 Nov 2024
Remove Symmetries to Control Model Expressivity and Improve Optimization Liu Ziyin Yizhou Xu Isaac Chuang AAML 98 4 0 28 Aug 2024
Connectivity Shapes Implicit Regularization in Matrix Factorization Models for Matrix Completion Zhiwei Bai Jiajie Zhao Yaoyu Zhang AI4CE 82 0 0 22 May 2024
Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding Zhengqing Wu Berfin Simsek Francois Ged ODL 111 0 0 08 Feb 2024
Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization Leyang Zhang Yaoyu Zhang Yaoyu Zhang 84 2 0 01 Sep 2023
Embedding Principle in Depth for the Loss Landscape Analysis of Deep Neural Networks Zhiwei Bai Yaoyu Zhang Z. Xu Yaoyu Zhang 91 6 0 26 May 2022
Noether: The More Things Change, the More Stay the Same Grzegorz Gluch R. Urbanke 48 18 0 12 Apr 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 157 80 0 08 Dec 2020
GENNI: Visualising the Geometry of Equivalences for Neural Network Identifiability Daniel Lengyel Janith C. Petangoda Isak Falk Kate Highnam Michalis Lazarou A. Kolbeinsson M. Deisenroth N. Jennings 59 5 0 14 Nov 2020
Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the Neural Tangent Kernel Stanislav Fort Gintare Karolina Dziugaite Mansheej Paul Sepideh Kharaghani Daniel M. Roy Surya Ganguli 109 193 0 28 Oct 2020
Finite Versus Infinite Neural Networks: an Empirical Study Jaehoon Lee S. Schoenholz Jeffrey Pennington Ben Adlam Lechao Xiao Roman Novak Jascha Narain Sohl-Dickstein 80 214 0 31 Jul 2020
The critical locus of overparameterized neural networks Y. Cooper UQCV 55 10 0 08 May 2020
Linear Mode Connectivity and the Lottery Ticket Hypothesis Jonathan Frankle Gintare Karolina Dziugaite Daniel M. Roy Michael Carbin MoMe 163 630 0 11 Dec 2019
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape Johanni Brea Berfin Simsek Bernd Illing W. Gerstner 98 58 0 05 Jul 2019
Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets Rohith Kuditipudi Xiang Wang Holden Lee Yi Zhang Zhiyuan Li Wei Hu Sanjeev Arora Rong Ge FAtt 113 93 0 14 Jun 2019
Semi-flat minima and saddle points by embedding neural networks to overparameterization Kenji Fukumizu Shoichiro Yamaguchi Yoh-ichi Mototake Mirai Tanaka 3DPC 54 25 0 12 Jun 2019
Large Scale Structure of Neural Network Loss Landscapes Stanislav Fort Stanislaw Jastrzebski 63 84 0 11 Jun 2019
Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent Jaehoon Lee Lechao Xiao S. Schoenholz Yasaman Bahri Roman Novak Jascha Narain Sohl-Dickstein Jeffrey Pennington 213 1,110 0 18 Feb 2019
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruosong Wang MLT 223 974 0 24 Jan 2019
On Connected Sublevel Sets in Deep Learning Quynh N. Nguyen 114 102 0 22 Jan 2019
Gradient Descent Finds Global Minima of Deep Neural Networks S. Du Jason D. Lee Haochuan Li Liwei Wang Masayoshi Tomizuka ODL 229 1,136 0 09 Nov 2018
Gradient Descent Provably Optimizes Over-parameterized Neural Networks S. Du Xiyu Zhai Barnabás Póczós Aarti Singh MLT ODL 243 1,276 0 04 Oct 2018
The jamming transition as a paradigm to understand the loss landscape of deep neural networks Mario Geiger S. Spigler Stéphane dÁscoli Levent Sagun Marco Baity-Jesi Giulio Biroli Matthieu Wyart 82 143 0 25 Sep 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 275 3,223 0 20 Jun 2018
On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport Lénaïc Chizat Francis R. Bach OT 221 737 0 24 May 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 274 3,488 0 09 Mar 2018
Essentially No Barriers in Neural Network Energy Landscape Felix Dräxler K. Veschgini M. Salmhofer Fred Hamprecht MoMe 122 435 0 02 Mar 2018
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs T. Garipov Pavel Izmailov Dmitrii Podoprikhin Dmitry Vetrov A. Wilson UQCV 95 758 0 27 Feb 2018
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks Levent Sagun Utku Evci V. U. Güney Yann N. Dauphin Léon Bottou 65 419 0 14 Jun 2017
Gradient Descent Can Take Exponential Time to Escape Saddle Points S. Du Chi Jin Jason D. Lee Michael I. Jordan Barnabás Póczós Aarti Singh 75 244 0 29 May 2017
How to Escape Saddle Points Efficiently Chi Jin Rong Ge Praneeth Netrapalli Sham Kakade Michael I. Jordan ODL 237 838 0 02 Mar 2017
Topology and Geometry of Half-Rectified Network Optimization C. Freeman Joan Bruna 222 235 0 04 Nov 2016
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.1K 150,364 0 22 Dec 2014
Explorations on high dimensional landscapes Levent Sagun V. U. Güney Gerard Ben Arous Yann LeCun 84 65 0 20 Dec 2014
Identifying and attacking the saddle point problem in high-dimensional non-convex optimization Yann N. Dauphin Razvan Pascanu Çağlar Gülçehre Kyunghyun Cho Surya Ganguli Yoshua Bengio ODL 134 1,394 0 10 Jun 2014