Are All Losses Created Equal: A Neural Collapse Perspective

4 October 2022

Sheng Liu

Qing Qu

Zhihui Zhu

ArXiv PDF HTML

Papers citing "Are All Losses Created Equal: A Neural Collapse Perspective"

50 / 52 papers shown

Title
Neural Multivariate Regression: Qualitative Insights from the Unconstrained Feature Model G. Andriopoulos Soyuj Jung Basnet Juan Guevara Li Guo Keith Ross 27 0 0 14 May 2025
On the Geometry of Semantics in Next-token Prediction Yize Zhao Christos Thrampoulidis 23 0 0 13 May 2025
A Good Start Matters: Enhancing Continual Learning with Data-Driven Weight Initialization Md Yousuf Harun Christopher Kanan AI4CE 50 0 0 09 Mar 2025
Category-free Out-of-Distribution Node Detection with Feature Resonance Shenzhi Yang Junbo Zhao Shouqing Yang Yixuan Li Dingyu Yang Xiaofang Zhang Haobo Wang OODD 51 0 0 22 Feb 2025
MaxSup: Overcoming Representation Collapse in Label Smoothing Yuxuan Zhou Heng Li Zhi-Qi Cheng Xudong Yan Mario Fritz M. Keuper 41 0 0 18 Feb 2025
Multi-layer Radial Basis Function Networks for Out-of-distribution Detection Amol Khanna Chenyi Ling Derek Everett Edward Raff Nathan Inkawhich OODD 28 0 0 05 Jan 2025
Guiding Neural Collapse: Optimising Towards the Nearest Simplex Equiangular Tight Frame Evan Markou Thalaiyasingam Ajanthan Stephen Gould 26 0 0 02 Nov 2024
The Persistence of Neural Collapse Despite Low-Rank Bias: An Analytic Perspective Through Unconstrained Features Connall Garrod Jonathan P. Keating 34 2 0 30 Oct 2024
$$\ell_1$-norm rank-one symmetric matrix factorization has no spurious second-order stationary points$ $\ell_1$ -norm rank-one symmetric matrix factorization has no spurious second-order stationary points Jiewen Guan Anthony Man-Cho So 39 2 0 07 Oct 2024
Wide Neural Networks Trained with Weight Decay Provably Exhibit Neural Collapse Arthur Jacot Peter Súkeník Zihan Wang Marco Mondelli 31 1 0 07 Oct 2024
Collapsed Language Models Promote Fairness Jingxuan Xu Wuyang Chen Linyi Li Yao Zhao Yunchao Wei 41 0 0 06 Oct 2024
Exploring Information-Theoretic Metrics Associated with Neural Collapse in Supervised Training Kun Song Zhiquan Tan Bochao Zou Jiansheng Chen Huimin Ma Weiran Huang 37 0 0 25 Sep 2024
The Prevalence of Neural Collapse in Neural Multivariate Regression G. Andriopoulos Zixuan Dong Li Guo Zifan Zhao Keith Ross 47 3 0 06 Sep 2024
Lightweight Uncertainty Quantification with Simplex Semantic Segmentation for Terrain Traversability Judith Dijk Gertjan J. Burghouts Kapil D. Katyal Bryanna Y. Yeh Craig Knuth Ella Fokkinga Tejaswi Kasarla Pascal Mettes 27 0 0 18 Jul 2024
Coding for Intelligence from the Perspective of Category Wenhan Yang Zixuan Hu Lilang Lin Jiaying Liu Ling-Yu Duan AI4CE 40 1 0 01 Jul 2024
Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation Can Yaras Peng Wang Laura Balzano Qing Qu AI4CE 37 12 0 06 Jun 2024
Unveiling the Dynamics of Information Interplay in Supervised Learning Kun Song Zhiquan Tan Bochao Zou Huimin Ma Weiran Huang 30 1 0 06 Jun 2024
Linguistic Collapse: Neural Collapse in (Large) Language Models Robert Wu V. Papyan 46 12 0 28 May 2024
Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal? Peter Súkeník Marco Mondelli Christoph H. Lampert AI4CE 32 7 0 23 May 2024
Navigate Beyond Shortcuts: Debiased Learning through the Lens of Neural Collapse Yining Wang Junjie Sun Chenyue Wang Mi Zhang Min Yang 27 6 0 09 May 2024
Adapting WavLM for Speech Emotion Recognition Daria Diatlova Anton Udalov Vitalii Shutov Egor Spirin 33 4 0 07 May 2024
Progressive Feedforward Collapse of ResNet Training Sicong Wang Kuo Gai Shihua Zhang AI4CE 33 4 0 02 May 2024
Unifying Low Dimensional Observations in Deep Learning Through the Deep Linear Unconstrained Feature Model Connall Garrod Jonathan P. Keating 33 8 0 09 Apr 2024
Investigating the Benefits of Projection Head for Representation Learning Yihao Xue Eric Gan Jiayi Ni Siddharth Joshi Baharan Mirzasoleiman 41 10 0 18 Mar 2024
Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features Tina Behnia Christos Thrampoulidis SSL 33 0 0 29 Feb 2024
Low-Rank Learning by Design: the Role of Network Architecture and Activation Linearity in Gradient Rank Collapse Bradley T. Baker Ba Pearlmutter Robyn L. Miller Vince D. Calhoun Sergey Plis AI4CE 11 2 0 09 Feb 2024
Pushing Boundaries: Mixup's Influence on Neural Collapse Quinn Fisher Haoming Meng V. Papyan AAML UQCV 36 5 0 09 Feb 2024
Cross Entropy versus Label Smoothing: A Neural Collapse Perspective Li Guo Keith Ross Zifan Zhao G. Andriopoulos Shuyang Ling Yufeng Xu Zixuan Dong UQCV NoLa 25 9 0 06 Feb 2024
Neural Collapse for Cross-entropy Class-Imbalanced Learning with Unconstrained ReLU Feature Model Hien Dang Tho Tran Tan Minh Nguyen Nhat Ho 15 11 0 04 Jan 2024
Convergence Analysis for Learning Orthonormal Deep Linear Neural Networks Zhen Qin Xuwei Tan Zhihui Zhu 32 0 0 24 Nov 2023
Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics Soo Min Kwon Zekai Zhang Dogyoon Song Laura Balzano Qing Qu 37 2 0 08 Nov 2023
Neural Collapse in Multi-label Learning with Pick-all-label Loss Pengyu Li Xiao Li Yutong Wang Qing Qu 22 8 0 24 Oct 2023
A Unified Approach to Domain Incremental Learning with Memory: Theory and Algorithm Haizhou Shi Hao Wang CLL 32 18 0 18 Oct 2023
Towards Demystifying the Generalization Behaviors When Neural Collapse Emerges Peifeng Gao Qianqian Xu Yibo Yang Peisong Wen Huiyang Shao Zhiyong Yang Bernard Ghanem Qingming Huang AAML 18 3 0 12 Oct 2023
Generalized Neural Collapse for a Large Number of Classes Jiachen Jiang Jinxin Zhou Peng Wang Qing Qu Dustin Mixon Chong You Zhihui Zhu AI4CE 32 23 0 09 Oct 2023
Neural Collapse for Unconstrained Feature Model under Cross-entropy Loss with Imbalanced Data Wanli Hong Shuyang Ling 21 14 0 18 Sep 2023
Neural Collapse in the Intermediate Hidden Layers of Classification Neural Networks Liam Parker Emre Onal Anton Stengel Jake Intrater 25 11 0 05 Aug 2023
Symmetric Neural-Collapse Representations with Supervised Contrastive Loss: The Impact of ReLU and Batching Ganesh Ramachandra Kini V. Vakilian Tina Behnia Jaidev Gill Christos Thrampoulidis 17 1 0 13 Jun 2023
Quantifying the Variability Collapse of Neural Networks Jing-Xue Xu Haoxiong Liu 33 4 0 06 Jun 2023
The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks Can Yaras P. Wang Wei Hu Zhihui Zhu Laura Balzano Qing Qu 33 17 0 01 Jun 2023
Matrix Information Theory for Self-Supervised Learning Yifan Zhang Zhi-Hao Tan Jingqin Yang Weiran Huang Yang Yuan SSL 45 16 0 27 May 2023
Which Features are Learnt by Contrastive Learning? On the Role of Simplicity Bias in Class Collapse and Feature Suppression Yihao Xue S. Joshi Eric Gan Pin-Yu Chen Baharan Mirzasoleiman SSL 30 23 0 25 May 2023
Deep Neural Collapse Is Provably Optimal for the Deep Unconstrained Features Model Peter Súkeník Marco Mondelli Christoph H. Lampert 26 21 0 22 May 2023
A Study of Neural Collapse Phenomenon: Grassmannian Frame, Symmetry and Generalization Peifeng Gao Qianqian Xu Peisong Wen Huiyang Shao Zhiyong Yang Qingming Huang 12 6 0 18 Apr 2023
On the Implicit Geometry of Cross-Entropy Parameterizations for Label-Imbalanced Data Tina Behnia Ganesh Ramachandra Kini V. Vakilian Christos Thrampoulidis 31 17 0 14 Mar 2023
Generalizing and Decoupling Neural Collapse via Hyperspherical Uniformity Gap Weiyang Liu L. Yu Adrian Weller Bernhard Schölkopf 32 17 0 11 Mar 2023
Unsupervised Manifold Linearizing and Clustering Tianjiao Ding Shengbang Tong Kwan Ho Ryan Chan Xili Dai Y. Ma B. Haeffele 16 11 0 04 Jan 2023
Principled and Efficient Transfer Learning of Deep Models via Neural Collapse Xiao Li Sheng Liu Jin-li Zhou Xin Lu C. Fernandez‐Granda Zhihui Zhu Q. Qu AAML 23 18 0 23 Dec 2022
Perturbation Analysis of Neural Collapse Tom Tirer Haoxiang Huang Jonathan Niles-Weed AAML 30 23 0 29 Oct 2022
Memorization-Dilation: Modeling Neural Collapse Under Label Noise Duc Anh Nguyen Ron Levie Julian Lienen Gitta Kutyniok Eyke Hüllermeier 16 1 0 11 Jun 2022