The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks

12 October 2021

Papers citing "The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks"

50 / 54 papers shown

Title
Understanding Mode Connectivity via Parameter Space Symmetry B. Zhao Nima Dehmamy Robin Walters Rose Yu 109 7 0 29 May 2025
Adiabatic Fine-Tuning of Neural Quantum States Enables Detection of Phase Transitions in Weight Space Vinicius Hernandes Thomas Spriggs Saqar Khaleefah E. Greplova 66 1 0 21 Mar 2025
SplatPose: Geometry-Aware 6-DoF Pose Estimation from Single RGB Image via 3D Gaussian Splatting Linqi Yang Xiongwei Zhao Qihao Sun Ke Wang Ao Chen Peng Kang 3DGS 98 3 0 07 Mar 2025
GNNMerge: Merging of GNN Models Without Accessing Training Data Vipul Garg Ishita Thakre Sayan Ranu MoMe 115 0 0 05 Mar 2025
Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Models Shuqi Liu Han Wu Bowei He Xiongwei Han Mingxuan Yuan Linqi Song MoMe 98 3 0 20 Feb 2025
High-dimensional manifold of solutions in neural networks: insights from statistical physics Enrico M. Malatesta 68 4 0 20 Feb 2025
Forget the Data and Fine-Tuning! Just Fold the Network to Compress Dong Wang Haris Šikić Lothar Thiele O. Saukh 79 1 0 17 Feb 2025
Linear Mode Connectivity in Differentiable Tree Ensembles Ryuichi Kanoh M. Sugiyama 134 1 0 17 Feb 2025
Beyond the Permutation Symmetry of Transformers: The Role of Rotation for Model Fusion Binchi Zhang Zaiyi Zheng Zhengzhang Chen Wenlin Yao 106 0 0 01 Feb 2025
Merging Feed-Forward Sublayers for Compressed Transformers Neha Verma Kenton W. Murray Kevin Duh AI4CE 86 0 0 10 Jan 2025
Training-free Heterogeneous Model Merging Zhengqi Xu Han Zheng Jie Song Li Sun Mingli Song MoMe 166 1 0 03 Jan 2025
ATM: Improving Model Merging by Alternating Tuning and Merging Luca Zhou Daniele Solombrino Donato Crisostomi Maria Sofia Bucarelli Fabrizio Silvestri Emanuele Rodolà MoMe 80 5 0 05 Nov 2024
Sampling from Bayesian Neural Network Posteriors with Symmetric Minibatch Splitting Langevin Dynamics Daniel Paulin Peter Whalley Neil K. Chada Benedict Leimkuhler BDL 67 4 0 14 Oct 2024
Remove Symmetries to Control Model Expressivity and Improve Optimization Liu Ziyin Yizhou Xu Isaac Chuang AAML 57 1 0 28 Aug 2024
Neural Networks Trained by Weight Permutation are Universal Approximators Yongqiang Cai Gaohang Chen Zhonghua Qiao 94 1 0 01 Jul 2024
Text-to-Model: Text-Conditioned Neural Network Diffusion for Train-Once-for-All Personalization Zexi Li Lingzhi Gao Chao Wu AI4CE DiffM 74 3 0 23 May 2024
Arcee's MergeKit: A Toolkit for Merging Large Language Models Charles Goddard Shamane Siriwardhana Malikeh Ehghaghi Luke Meyers Vladimir Karpukhin Brian Benedict Mark McQuade Jacob Solawetz MoMe KELM 112 92 0 20 Mar 2024
MedMerge: Merging Models for Effective Transfer Learning to Medical Imaging Tasks Ibrahim Almakky Santosh Sanjeev Anees Ur Rehman Hashmi Mohammad Areeb Qazi Mohammad Yaqub Mohammad Yaqub FedML MoMe 113 4 0 18 Mar 2024
Neural Redshift: Random Networks are not Random Functions Damien Teney A. Nicolicioiu Valentin Hartmann Ehsan Abbasnejad 117 22 0 04 Mar 2024
Analysis of Linear Mode Connectivity via Permutation-Based Weight Matching: With Insights into Other Permutation Search Methods Akira Ito Masanori Yamada Atsutoshi Kumagai MoMe 83 5 0 06 Feb 2024
Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances Berfin cSimcsek François Ged Arthur Jacot Francesco Spadaro Clément Hongler W. Gerstner Johanni Brea AI4CE 61 95 0 25 May 2021
Learning Neural Network Subspaces Mitchell Wortsman Maxwell Horton Carlos Guestrin Ali Farhadi Mohammad Rastegari UQCV 41 85 0 20 Feb 2021
Optimizing Mode Connectivity via Neuron Alignment N. Joseph Tatro Pin-Yu Chen Payel Das Igor Melnyk P. Sattigeri Rongjie Lai MoMe 256 81 0 05 Sep 2020
What is being transferred in transfer learning? Behnam Neyshabur Hanie Sedghi Chiyuan Zhang 62 513 0 26 Aug 2020
Towards Learning Convolutions from Scratch Behnam Neyshabur SSL 257 71 0 27 Jul 2020
Loss landscapes and optimization in over-parameterized non-linear systems and neural networks Chaoyue Liu Libin Zhu M. Belkin ODL 39 258 0 29 Feb 2020
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning Yeming Wen Dustin Tran Jimmy Ba OOD FedML UQCV 103 487 0 17 Feb 2020
Linear Mode Connectivity and the Lottery Ticket Hypothesis Jonathan Frankle Gintare Karolina Dziugaite Daniel M. Roy Michael Carbin MoMe 125 611 0 11 Dec 2019
Deep Ensembles: A Loss Landscape Perspective Stanislav Fort Huiyi Hu Balaji Lakshminarayanan OOD UQCV 65 624 0 05 Dec 2019
Deep Double Descent: Where Bigger Models and More Data Hurt Preetum Nakkiran Gal Kaplun Yamini Bansal Tristan Yang Boaz Barak Ilya Sutskever 97 925 0 04 Dec 2019
Implicit Regularization and Convergence for Weight Normalization Xiaoxia Wu Yan Sun Tongzheng Ren Shanshan Wu Zhiyuan Li Suriya Gunasekar Rachel A. Ward Qiang Liu 118 21 0 18 Nov 2019
Model Fusion via Optimal Transport Sidak Pal Singh Martin Jaggi MoMe FedML 72 231 0 12 Oct 2019
Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape Johanni Brea Berfin Simsek Bernd Illing W. Gerstner 69 55 0 05 Jul 2019
Large Scale Structure of Neural Network Loss Landscapes Stanislav Fort Stanislaw Jastrzebski 39 83 0 11 Jun 2019
Shaping the learning landscape in neural networks around wide flat minima Carlo Baldassi Fabrizio Pittorino R. Zecchina MLT 36 82 0 20 May 2019
Uniform convergence may be unable to explain generalization in deep learning Vaishnavh Nagarajan J. Zico Kolter MoMe AI4CE 44 314 0 13 Feb 2019
Reconciling modern machine learning practice and the bias-variance trade-off M. Belkin Daniel J. Hsu Siyuan Ma Soumik Mandal 168 1,628 0 28 Dec 2018
On the loss landscape of a class of deep neural networks with no bad local valleys Quynh N. Nguyen Mahesh Chandra Mukkamala Matthias Hein 47 87 0 27 Sep 2018
The jamming transition as a paradigm to understand the loss landscape of deep neural networks Mario Geiger S. Spigler Stéphane dÁscoli Levent Sagun Marco Baity-Jesi Giulio Biroli Matthieu Wyart 46 141 0 25 Sep 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 162 3,160 0 20 Jun 2018
Multi-Task Zipping via Layer-wise Neuron Sharing Xiaoxi He Zimu Zhou Lothar Thiele MoMe 20 62 0 24 May 2018
A Mean Field View of the Landscape of Two-Layers Neural Networks Song Mei Andrea Montanari Phan-Minh Nguyen MLT 69 855 0 18 Apr 2018
Averaging Weights Leads to Wider Optima and Better Generalization Pavel Izmailov Dmitrii Podoprikhin T. Garipov Dmitry Vetrov A. Wilson FedML MoMe 93 1,643 0 14 Mar 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 164 3,433 0 09 Mar 2018
Essentially No Barriers in Neural Network Energy Landscape Felix Dräxler K. Veschgini M. Salmhofer Fred Hamprecht MoMe 97 430 0 02 Mar 2018
Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs T. Garipov Pavel Izmailov Dmitrii Podoprikhin Dmitry Vetrov A. Wilson UQCV 56 746 0 27 Feb 2018
Visualizing the Loss Landscape of Neural Nets Hao Li Zheng Xu Gavin Taylor Christoph Studer Tom Goldstein 230 1,873 0 28 Dec 2017
Exploring Generalization in Deep Learning Behnam Neyshabur Srinadh Bhojanapalli David A. McAllester Nathan Srebro FAtt 132 1,245 0 27 Jun 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 262 4,612 0 10 Nov 2016
Topology and Geometry of Half-Rectified Network Optimization C. Freeman Joan Bruna 131 235 0 04 Nov 2016