Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!

3 March 2023

Papers citing "Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together!"

33 / 33 papers shown

Title
Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing Qi Le Enmao Diao Ziyan Wang Xinran Wang Jie Ding Li Yang Ali Anwar 69 1 0 24 Feb 2025
Sparsity May Be All You Need: Sparse Random Parameter Adaptation Jesus Rios Pierre L. Dognin Ronny Luss K. Ramamurthy 32 1 0 21 Feb 2025
Navigating Extremes: Dynamic Sparsity in Large Output Spaces Nasib Ullah Erik Schultheis Mike Lasby Yani Andrew Ioannou Rohit Babbar 33 0 0 05 Nov 2024
Are Sparse Neural Networks Better Hard Sample Learners? Q. Xiao Boqian Wu Lu Yin Christopher Neil Gadzinski Tianjin Huang Mykola Pechenizkiy D. Mocanu 37 1 0 13 Sep 2024
From GaLore to WeLore: How Low-Rank Weights Non-uniformly Emerge from Low-Rank Gradients Ajay Jaiswal Lu Yin Zhenyu (Allen) Zhang Shiwei Liu Jiawei Zhao Yuandong Tian Zhangyang Wang 33 14 0 15 Jul 2024
Sparsest Models Elude Pruning: An Exposé of Pruning's Current Capabilities Stephen Zhang V. Papyan 28 0 0 04 Jul 2024
FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skipping Ajay Jaiswal Bodun Hu Lu Yin Yeonju Ro Shiwei Liu Tianlong Chen Aditya Akella 53 12 0 05 Apr 2024
Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compression Junyuan Hong Jinhao Duan Chenhui Zhang Zhangheng Li Chulin Xie ... B. Kailkhura Dan Hendrycks Dawn Song Zhangyang Wang Bo-wen Li 36 24 0 18 Mar 2024
Random Search as a Baseline for Sparse Neural Network Architecture Search Rezsa Farahani 16 0 0 13 Mar 2024
Sparsity-Guided Holistic Explanation for LLMs with Interpretable Inference-Time Intervention Zhen Tan Tianlong Chen Zhenyu (Allen) Zhang Huan Liu 44 14 0 22 Dec 2023
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark Eldar Kurtic Torsten Hoefler Dan Alistarh 29 3 0 21 Dec 2023
Visual Prompting Upgrades Neural Network Sparsification: A Data-Model Perspective Can Jin Tianjin Huang Yihua Zhang Mykola Pechenizkiy Sijia Liu Shiwei Liu Tianlong Chen VLM 25 26 0 03 Dec 2023
Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs Yu-xin Zhang Lirui Zhao Mingbao Lin Yunyun Sun Yiwu Yao Xingjia Han Jared Tanner Shiwei Liu Rongrong Ji SyDa 37 40 0 13 Oct 2023
Compressing LLMs: The Truth is Rarely Pure and Never Simple Ajay Jaiswal Zhe Gan Xianzhi Du Bowen Zhang Zhangyang Wang Yinfei Yang MQ 36 45 0 02 Oct 2023
Pruning Small Pre-Trained Weights Irreversibly and Monotonically Impairs "Difficult" Downstream Tasks in LLMs Lu Yin Ajay Jaiswal Shiwei Liu Souvik Kundu Zhangyang Wang 22 7 0 29 Sep 2023
Unveiling Invariances via Neural Network Pruning Derek Xu Yizhou Sun Wei Wang 36 0 0 15 Sep 2023
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training A. Nowak Bram Grooten D. Mocanu Jacek Tabor 21 9 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 56 353 0 20 Jun 2023
Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication A. Jaiswal Shiwei Liu Tianlong Chen Ying Ding Zhangyang Wang GNN 36 5 0 18 Jun 2023
Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Models A. Jaiswal Shiwei Liu Tianlong Chen Ying Ding Zhangyang Wang VLM 32 22 0 18 Jun 2023
The Emergence of Essential Sparsity in Large Pre-trained Models: The Weights that Matter Ajay Jaiswal Shiwei Liu Tianlong Chen Zhangyang Wang VLM 21 33 0 06 Jun 2023
Dense Sparse Retrieval: Using Sparse Language Models for Inference Efficient Dense Retrieval Daniel Fernando Campos ChengXiang Zhai 18 0 0 31 Mar 2023
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models Vithursan Thangarasa Abhay Gupta William Marshall Tianda Li Kevin Leong D. DeCoste Sean Lie Shreyas Saxena MoE AI4CE 16 18 0 18 Mar 2023
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers Shiwei Liu Zhangyang Wang 30 30 0 06 Feb 2023
A Kernel-Based View of Language Model Fine-Tuning Sadhika Malladi Alexander Wettig Dingli Yu Danqi Chen Sanjeev Arora VLM 68 60 0 11 Oct 2022
Sparsity Winning Twice: Better Robust Generalization from More Efficient Training Tianlong Chen Zhenyu (Allen) Zhang Pengju Wang Santosh Balachandra Haoyu Ma Zehao Wang Zhangyang Wang OOD AAML 77 46 0 20 Feb 2022
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 355 8,457 0 28 Jan 2022
Towards More Effective and Economic Sparsely-Activated Model Hao Jiang Ke Zhan Jianwei Qu Yongkang Wu Zhaoye Fei ... Enrui Hu Yinxia Zhang Yantao Jia Fan Yu Zhao Cao MoE 147 12 0 14 Oct 2021
Powerpropagation: A sparsity inducing weight reparameterisation Jonathan Richard Schwarz Siddhant M. Jayakumar Razvan Pascanu P. Latham Yee Whye Teh 87 54 0 01 Oct 2021
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 150 345 0 23 Jul 2020
What is the State of Neural Network Pruning? Davis W. Blalock Jose Javier Gonzalez Ortiz Jonathan Frankle John Guttag 185 1,027 0 06 Mar 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 222 382 0 05 Mar 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,956 0 20 Apr 2018