Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks

16 December 2020

Xiangyu Chang

Yingcong Li

Samet Oymak

Christos Thrampoulidis

ArXiv PDF HTML

Papers citing "Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks"

34 / 34 papers shown

Title
MergeBench: A Benchmark for Merging Domain-Specialized LLMs Yifei He Siqi Zeng Yuzheng Hu Rui Yang Tong Zhang Han Zhao MoMe ALM 19 0 0 16 May 2025
Spurious Correlations in High Dimensional Regression: The Roles of Regularization, Simplicity Bias and Over-Parameterization Simone Bombari Marco Mondelli 124 0 0 03 Feb 2025
Accelerating Linear Recurrent Neural Networks for the Edge with Unstructured Sparsity Alessandro Pierro Steven Abreu Jonathan Timcheck Philipp Stratmann Andreas Wild S. Shrestha 70 0 0 03 Feb 2025
Implicit Bias in Matrix Factorization and its Explicit Realization in a New Architecture Yikun Hou Suvrit Sra A. Yurtsever 31 0 0 28 Jan 2025
Beyond adaptive gradient: Fast-Controlled Minibatch Algorithm for large-scale optimization Corrado Coppola Lorenzo Papa Irene Amerini L. Palagi ODL 79 0 0 24 Nov 2024
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws M. E. Ildiz Halil Alperen Gozeten Ege Onur Taga Marco Mondelli Samet Oymak 54 2 0 24 Oct 2024
Precise asymptotics of reweighted least-squares algorithms for linear diagonal networks Chiraag Kaushik Justin Romberg Vidya Muthukumar 39 1 0 04 Jun 2024
Occam Gradient Descent B. N. Kausik ODL VLM 32 0 0 30 May 2024
Class-wise Activation Unravelling the Engima of Deep Double Descent Yufei Gu 36 0 0 13 May 2024
Masks, Signs, And Learning Rate Rewinding Advait Gadhikar R. Burkholz 55 8 0 29 Feb 2024
Understanding the Role of Optimization in Double Descent Chris Liu Jeffrey Flanigan 32 0 0 06 Dec 2023
Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics Soo Min Kwon Zekai Zhang Dogyoon Song Laura Balzano Qing Qu 39 2 0 08 Nov 2023
Unraveling the Enigma of Double Descent: An In-depth Analysis through the Lens of Learned Feature Space Yufei Gu Xiaoqing Zheng T. Aste 37 3 0 20 Oct 2023
The Quest of Finding the Antidote to Sparse Double Descent Victor Quétu Marta Milovanović 31 0 0 31 Aug 2023
DSD $^2$ : Can We Dodge Sparse Double Descent and Compress the Neural Network Worry-Free? Victor Quétu Enzo Tartaglione 29 7 0 02 Mar 2023
Can we avoid Double Descent in Deep Neural Networks? Victor Quétu Enzo Tartaglione AI4CE 20 3 0 26 Feb 2023
Precise Asymptotic Analysis of Deep Random Feature Models David Bosch Ashkan Panahi B. Hassibi 35 19 0 13 Feb 2023
Strong inductive biases provably prevent harmless interpolation Michael Aerni Marco Milanta Konstantin Donhauser Fanny Yang 32 9 0 18 Jan 2023
Why Random Pruning Is All We Need to Start Sparse Advait Gadhikar Sohom Mukherjee R. Burkholz 43 19 0 05 Oct 2022
Deep Double Descent via Smooth Interpolation Matteo Gamba Erik Englesson Marten Bjorkman Hossein Azizpour 63 10 0 21 Sep 2022
Overparameterization from Computational Constraints Sanjam Garg S. Jha Saeed Mahloujifar Mohammad Mahmoody Mingyuan Wang 20 1 0 27 Aug 2022
Sparse Double Descent: Where Network Pruning Aggravates Overfitting Zhengqi He Zeke Xie Quanzhi Zhu Zengchang Qin 69 27 0 17 Jun 2022
Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis Wuyang Chen Wei Huang Xinyu Gong Boris Hanin Zhangyang Wang 30 7 0 11 May 2022
Random Features Model with General Convex Regularization: A Fine Grained Analysis with Precise Asymptotic Learning Curves David Bosch Ashkan Panahi Ayça Özçelikkale Devdatt Dubhash MLT 16 2 0 06 Apr 2022
Provable and Efficient Continual Representation Learning Yingcong Li Mingchen Li M. Salman Asif Samet Oymak CLL 30 11 0 03 Mar 2022
Towards Sample-efficient Overparameterized Meta-learning Yue Sun Adhyyan Narang Halil Ibrahim Gulluk Samet Oymak Maryam Fazel BDL 30 24 0 16 Jan 2022
A Farewell to the Bias-Variance Tradeoff? An Overview of the Theory of Overparameterized Machine Learning Yehuda Dar Vidya Muthukumar Richard G. Baraniuk 29 71 0 06 Sep 2021
How much pre-training is enough to discover a good subnetwork? Cameron R. Wolfe Fangshuo Liao Qihan Wang J. Kim Anastasios Kyrillidis 30 3 0 31 Jul 2021
Spectral Pruning for Recurrent Neural Networks Takashi Furuya Kazuma Suetake K. Taniguchi Hiroyuki Kusumoto Ryuji Saiin Tomohiro Daimon 27 4 0 23 May 2021
Generalization Guarantees for Neural Architecture Search with Train-Validation Split Samet Oymak Mingchen Li Mahdi Soltanolkotabi AI4CE OOD 36 13 0 29 Apr 2021
Lottery Jackpots Exist in Pre-trained Models Yu-xin Zhang Mingbao Lin Yan Wang Rongrong Ji Rongrong Ji 30 15 0 18 Apr 2021
Label-Imbalanced and Group-Sensitive Classification under Overparameterization Ganesh Ramachandra Kini Orestis Paraskevas Samet Oymak Christos Thrampoulidis 27 93 0 02 Mar 2021
Distilling Double Descent Andrew Cotter A. Menon Harikrishna Narasimhan A. S. Rawat Sashank J. Reddi Yichen Zhou 25 7 0 13 Feb 2021
Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularization Ke Wang Christos Thrampoulidis 28 27 0 18 Nov 2020