Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers

21 May 2025

Papers citing "Neural Collapse is Globally Optimal in Deep Regularized ResNets and Transformers"

50 / 53 papers shown

Title
Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations Yize Zhao Tina Behnia V. Vakilian Christos Thrampoulidis 176 10 0 20 Feb 2025
On the inductive bias of infinite-depth ResNets and the bottleneck rank Enric Boix-Adsera 36 1 0 31 Jan 2025
The Exploration of Neural Collapse under Imbalanced Data Haixia Liu 83 3 0 26 Nov 2024
The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features Connall Garrod Jonathan P. Keating 58 4 0 30 Oct 2024
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse Arthur Jacot Peter Súkeník Zihan Wang Marco Mondelli 66 2 0 07 Oct 2024
The Prevalence of Neural Collapse in Neural Multivariate Regression Shuyang Ling Zixuan Dong Li Guo Zifan Zhao George Andriopoulos 70 7 0 06 Sep 2024
Beyond Unconstrained Features: Neural Collapse for Shallow Neural Networks with General Data Wanli Hong Shuyang Ling 62 4 0 03 Sep 2024
Linguistic Collapse: Neural Collapse in (Large) Language Models Robert Wu Vardan Papyan 87 16 0 28 May 2024
Pursuing Feature Separation based on Neural Collapse for Out-of-Distribution Detection Yingwen Wu Ruiji Yu Xinwen Cheng Zhengbao He Xiaolin Huang OODD 98 4 0 28 May 2024
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal? Peter Súkeník Marco Mondelli Christoph H. Lampert AI4CE 65 11 0 23 May 2024
Progressive Feedforward Collapse of ResNet Training Sicong Wang Kuo Gai Shihua Zhang AI4CE 64 6 0 02 May 2024
Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model Connall Garrod Jonathan P. Keating 93 9 0 09 Apr 2024
Average gradient outer product as a mechanism for deep neural collapse Daniel Beaglehole Peter Súkeník Marco Mondelli Misha Belkin AI4CE 67 13 0 21 Feb 2024
Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model Hien Dang Tho Tran Tan Minh Nguyen Nhat Ho 48 14 0 04 Jan 2024
EPA: Neural Collapse Inspired Robust Out-of-Distribution Detector Jiawei Zhang Yufan Chen Cheng Jin Lei Zhu Yuantao Gu OODD UQCV AAML 66 8 0 03 Jan 2024
On the Robustness of Neural Collapse and the Neural Collapse of Robustness Jingtong Su Ya Shi Zhang Nikolaos Tsilivis Julia Kempe AAML 59 7 0 13 Nov 2023
Generalized Neural Collapse for a Large Number of Classes Jiachen Jiang Jinxin Zhou Peng Wang Qing Qu Dustin Mixon Chong You Zhihui Zhu AI4CE 73 28 0 09 Oct 2023
Spurious Feature Diversification Improves Out-of-distribution Generalization Yong Lin Lu Tan Yifan Hao Honam Wong Hanze Dong Weizhong Zhang Yujiu Yang Tong Zhang OODD 70 27 0 29 Sep 2023
Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data Wanli Hong Shuyang Ling 53 18 0 18 Sep 2023
Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representation Haoqi Wang Zhizhong Li Wayne Zhang 47 4 0 03 Aug 2023
A Neural Collapse Perspective on Feature Evolution in Graph Neural Networks Vignesh Kothapalli Tom Tirer Joan Bruna 76 13 0 04 Jul 2023
The Tunnel Effect: Building Data Representations in Deep Neural Networks Wojciech Masarczyk M. Ostaszewski Ehsan Imani Razvan Pascanu Piotr Milo's Tomasz Trzciñski 82 25 0 31 May 2023
How Does Information Bottleneck Help Deep Learning? Kenji Kawaguchi Zhun Deng Xu Ji Jiaoyang Huang 69 61 0 30 May 2023
Neural (Tangent Kernel) Collapse Mariia Seleznova Dana Weitzner Raja Giryes Gitta Kutyniok H. Chou 52 10 0 25 May 2023
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model Peter Súkeník Marco Mondelli Christoph H. Lampert 56 27 0 22 May 2023
No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifier Zexi Li Xinyi Shang Rui He Tao R. Lin Chao Wu FedML 91 55 0 17 Mar 2023
Inducing Neural Collapse to a Fixed Hierarchy-Aware Frame for Reducing Mistake Severity Tong Liang Jim Davis 97 11 0 10 Mar 2023
Inducing Neural Collapse in Deep Long-tailed Learning Xuantong Liu Jianfeng Zhang Tianyang Hu He Cao Lujia Pan Yuan Yao 58 31 0 24 Feb 2023
Understanding Imbalanced Semantic Segmentation Through Neural Collapse Zhisheng Zhong Jiequan Cui Yibo Yang Xiaoyang Wu Xiaojuan Qi Xinming Zhang Jiaya Jia 188 50 0 03 Jan 2023
Neural Collapse in Deep Linear Networks: From Balanced to Imbalanced Data Hien Dang Tho Tran T. Nguyen Hung The Tran Nhat Ho Hung Tran 93 32 0 01 Jan 2023
A Law of Data Separation in Deep Learning Hangfeng He Weijie J. Su OOD 78 41 0 31 Oct 2022
Perturbation Analysis of Neural Collapse Tom Tirer Haoxiang Huang Jonathan Niles-Weed AAML 79 26 0 29 Oct 2022
The Asymmetric Maximum Margin Bias of Quasi-Homogeneous Neural Networks D. Kunin Atsushi Yamamura Chao Ma Surya Ganguli 62 21 0 07 Oct 2022
Are All Losses Created Equal: A Neural Collapse Perspective Jinxin Zhou Chong You Xiao Li Kangning Liu Sheng Liu Qing Qu Zhihui Zhu 72 65 0 04 Oct 2022
Linking Neural Collapse and L2 Normalization with Improved Out-of-Distribution Detection in Deep Neural Networks J. Haas William Yolland B. Rabus OODD 75 20 0 17 Sep 2022
Improving Self-Supervised Learning by Characterizing Idealized Representations Yann Dubois Tatsunori Hashimoto Stefano Ermon Percy Liang SSL 111 43 0 13 Sep 2022
Imbalance Trouble: Revisiting Neural-Collapse Geometry Christos Thrampoulidis Ganesh Ramachandra Kini V. Vakilian Tina Behnia 58 73 0 10 Aug 2022
Balanced Contrastive Learning for Long-Tailed Visual Recognition Jianggang Zhu Ziyi Wang Jingjing Chen Yi-Ping Phoebe Chen Yueping Jiang 75 181 0 19 Jul 2022
Perfectly Balanced: Improving Transfer and Robustness of Supervised Contrastive Learning Mayee F. Chen Daniel Y. Fu A. Narayan Michael Zhang Zhao Song Kayvon Fatahalian Christopher Ré SSL 62 51 0 15 Apr 2022
On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features Jinxin Zhou Xiao Li Tian Ding Chong You Qing Qu Zhihui Zhu 72 102 0 02 Mar 2022
On the Implicit Bias Towards Minimal Depth of Deep Neural Networks Tomer Galanti Liane Galanti Ido Ben-Shaul 31 14 0 18 Feb 2022
Limitations of Neural Collapse for Understanding Generalization in Deep Learning Like Hui M. Belkin Preetum Nakkiran AI4CE 44 56 0 17 Feb 2022
Extended Unconstrained Features Model for Exploring Deep Neural Collapse Tom Tirer Joan Bruna AAML 48 101 0 16 Feb 2022
An Unconstrained Layer-Peeled Perspective on Neural Collapse Wenlong Ji Yiping Lu Yiliang Zhang Zhun Deng Weijie J. Su 178 87 0 06 Oct 2021
Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path X. Y. Han Vardan Papyan D. Donoho AAML 88 144 0 03 Jun 2021
A Geometric Analysis of Neural Collapse with Unconstrained Features Zhihui Zhu Tianyu Ding Jinxin Zhou Xiao Li Chong You Jeremias Sulam Qing Qu 69 205 0 06 May 2021
Exploring Deep Neural Networks via Layer-Peeled Model: Minority Collapse in Imbalanced Training Cong Fang Hangfeng He Qi Long Weijie J. Su FAtt 170 172 0 29 Jan 2021
On the emergence of simplex symmetry in the final and penultimate layers of neural network classifiers E. Weinan Stephan Wojtowytsch 76 45 0 10 Dec 2020
Neural collapse with unconstrained features D. Mixon Hans Parshall Jianzong Pi 72 121 0 23 Nov 2020
Prevalence of Neural Collapse during the terminal phase of deep learning training Vardan Papyan Xuemei Han D. Donoho 205 582 0 18 Aug 2020