Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

31 January 2021

Dan Alistarh

Papers citing "Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks"

50 / 136 papers shown

Title
Efficient Unstructured Pruning of Mamba State-Space Models for Resource-Constrained Environments Ibne Farabi Shihab Sanjeda Akter Anuj Sharma Mamba 50 0 0 13 May 2025
ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks Wenhao Hu Paul Henderson José Cano 32 0 0 12 May 2025
Onboard Optimization and Learning: A Survey Monirul Islam Pavel Siyi Hu Mahardhika Pratama Ryszard Kowalczyk 26 0 0 07 May 2025
Sparsity is All You Need: Rethinking Biological Pathway-Informed Approaches in Deep Learning Isabella Caranzano Corrado Pancotti Cesare Rollo Flavio Sartori Pietro Liò P. Fariselli Tiziana Sanavia OOD UQCV 65 0 0 07 May 2025
Periodic Online Testing for Sparse Systolic Tensor Arrays C. Peltekis Chrysostomos Nicopoulos G. Dimitrakopoulos 52 0 0 25 Apr 2025
Switch-Based Multi-Part Neural Network Surajit Majumder Paritosh Ranjan Prodip Roy Bhuban Padhan OOD 79 0 0 25 Apr 2025
Efficient FPGA-accelerated Convolutional Neural Networks for Cloud Detection on CubeSats Angela Cratere M. Salim Farissi Andrea Carbone Marcello Asciolla Maria Rizzi Francesco DellÓlio Augusto Nascetti Dario Spiller 28 1 0 04 Apr 2025
Towards Symmetric Low-Rank Adapters Tales Panoutsos Rodrygo L. T. Santos Flavio Figueiredo 33 0 0 29 Mar 2025
LapSum -- One Method to Differentiate Them All: Ranking, Sorting and Top-k Selection Łukasz Struski Michał B. Bednarczyk Igor T. Podolak Jacek Tabor BDL 62 0 0 08 Mar 2025
E2ENet: Dynamic Sparse Feature Fusion for Accurate and Efficient 3D Medical Image Segmentation Boqian Wu Q. Xiao Shiwei Liu Lu Yin Mykola Pechenizkiy Decebal Constantin Mocanu M. V. Keulen Elena Mocanu MedIm 65 4 0 20 Feb 2025
CondensNet: Enabling stable long-term climate simulations via hybrid deep learning models with adaptive physical constraints Xin Wang Juntao Yang Jeff Adie Simon See Kalli Furtado Chen Chen T. Arcomano R. Maulik G. Mengaldo AI4CE 52 0 0 18 Feb 2025
An Efficient Row-Based Sparse Fine-Tuning Cen-Jhih Li Aditya Bhaskara 56 0 0 17 Feb 2025
Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries Chris Kolb T. Weber Bernd Bischl David Rügamer 113 0 0 04 Feb 2025
Symmetric Pruning of Large Language Models Kai Yi Peter Richtárik AAML VLM 73 0 0 31 Jan 2025
Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models J. P. Muñoz Jinjie Yuan Nilesh Jain Mamba 72 1 0 28 Jan 2025
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 82 5 0 28 Jan 2025
UPAQ: A Framework for Real-Time and Energy-Efficient 3D Object Detection in Autonomous Vehicles Abhishek Balasubramaniam Febin P. Sunny S. Pasricha 3DPC 44 0 0 08 Jan 2025
Navigating Extremes: Dynamic Sparsity in Large Output Spaces Nasib Ullah Erik Schultheis Mike Lasby Yani Andrew Ioannou Rohit Babbar 35 0 0 05 Nov 2024
More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing Sagi Shaier Francisco Pereira K. Wense Lawrence E Hunter Matt Jones MoE 46 0 0 10 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 56 6 0 09 Oct 2024
LoRTA: Low Rank Tensor Adaptation of Large Language Models Ignacio Hounie Charilaos I. Kanatsoulis Arnuv Tandon Alejandro Ribeiro 36 0 0 05 Oct 2024
Efficient Source-Free Time-Series Adaptation via Parameter Subspace Disentanglement Gaurav Patel Christopher Sandino Behrooz Mahasseni Ellen L. Zippi Erdrin Azemi Ali Moin Juri Minxha TTA AI4TS 50 3 0 03 Oct 2024
EntryPrune: Neural Network Feature Selection using First Impressions Felix Zimmer Patrik Okanovic Torsten Hoefler 29 0 0 03 Oct 2024
Two Sparse Matrices are Better than One: Sparsifying Neural Networks with Double Sparse Factorization Vladimír Boža Vladimír Macko 30 1 0 27 Sep 2024
CRoP: Context-wise Robust Static Human-Sensing Personalization Sawinder Kaur Avery Gump Yi Xiao Jingyu Xin Harshit Sharma Nina R Benway Jonathan L Preston Asif Salekin 29 0 0 26 Sep 2024
Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts Xiaoming Shi Shiyu Wang Yuqi Nie Dianqi Li Zhou Ye Qingsong Wen Ming Jin AI4TS 46 28 0 24 Sep 2024
Self-Masking Networks for Unsupervised Adaptation Alfonso Taboada Warmerdam Mathilde Caron Yuki M. Asano 49 1 0 11 Sep 2024
On the Complexity of Neural Computation in Superposition Micah Adler Nir Shavit 123 3 0 05 Sep 2024
Network Fission Ensembles for Low-Cost Self-Ensembles Hojung Lee Jong-Seok Lee UQCV 64 0 0 05 Aug 2024
Compact Language Models via Pruning and Knowledge Distillation Saurav Muralidharan Sharath Turuvekere Sreenivas Raviraj Joshi Marcin Chochowski M. Patwary M. Shoeybi Bryan Catanzaro Jan Kautz Pavlo Molchanov SyDa MQ 44 38 0 19 Jul 2024
Learning Interpretable Differentiable Logic Networks Chang Yue N. Jha NAI AI4CE 29 0 0 04 Jul 2024
A Thorough Performance Benchmarking on Lightweight Embedding-based Recommender Systems Hung Vinh Tran Tong Chen Quoc Viet Hung Nguyen Zi-Rui Huang Lizhen Cui Hongzhi Yin 45 1 0 25 Jun 2024
Group Projected Subspace Pursuit for Block Sparse Signal Reconstruction: Convergence Analysis and Applications Roy Y. He Haixia Liu Hao Liu 23 2 0 01 Jun 2024
Effective Interplay between Sparsity and Quantization: From Theory to Practice Simla Burcu Harma Ayan Chakraborty Elizaveta Kostenok Danila Mishin Dongho Ha ... Martin Jaggi Ming Liu Yunho Oh Suvinay Subramanian Amir Yazdanbakhsh MQ 44 6 0 31 May 2024
$Dual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization$ Dual sparse training framework: inducing activation map sparsity via Transformed $\ell1$ regularization Xiaolong Yu Cong Tian 52 0 0 30 May 2024
Scorch: A Library for Sparse Deep Learning Bobby Yan Alexander J. Root Trevor Gale David Broman Fredrik Kjolstad 33 0 0 27 May 2024
Fast and Controllable Post-training Sparsity: Learning Optimal Sparsity Allocation with Global Constraint in Minutes Ruihao Gong Yang Yong Zining Wang Jinyang Guo Xiuying Wei Yuqing Ma Xianglong Liu 54 5 0 09 May 2024
Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment Abhinav Agarwalla Abhay Gupta Alexandre Marques Shubhra Pandit Michael Goin ... Tuan Nguyen Mahmoud Salem Dan Alistarh Sean Lie Mark Kurtz MoE SyDa 42 11 0 06 May 2024
Towards Green AI: Current status and future research Christian Clemm Lutz Stobbe Kishan Wimalawarne Jan Druschke 49 2 0 01 May 2024
Decentralized Personalized Federated Learning based on a Conditional Sparse-to-Sparser Scheme Qianyu Long Qiyuan Wang Christos Anagnostopoulos Daning Bi FedML 28 0 0 24 Apr 2024
Low-Depth Spatial Tree Algorithms Yves Baumann Tal Ben-Nun Maciej Besta Lukas Gianinazzi Torsten Hoefler Piotr Luczynski 42 0 0 19 Apr 2024
SparseDM: Toward Sparse Efficient Diffusion Models Kafeng Wang Jianfei Chen He Li Zhenpeng Mi Jun-Jie Zhu DiffM 68 8 0 16 Apr 2024
Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN Biswadeep Chakraborty Beomseok Kang H. Kumar Saibal Mukhopadhyay 46 8 0 06 Mar 2024
SparseLLM: Towards Global Pruning for Pre-trained Language Models Guangji Bai Yijiang Li Chen Ling Kibaek Kim Liang Zhao 33 6 0 28 Feb 2024
Always-Sparse Training by Growing Connections with Guided Stochastic Exploration Mike Heddes Narayan Srinivasa T. Givargis Alexandru Nicolau 91 0 0 12 Jan 2024
EsaCL: Efficient Continual Learning of Sparse Models Weijieying Ren V. Honavar CLL 28 3 0 11 Jan 2024
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Max Zimmer Megi Andoni Christoph Spiegel Sebastian Pokutta VLM 52 10 0 23 Dec 2023
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction Pratyusha Sharma Jordan T. Ash Dipendra Kumar Misra LRM 19 78 0 21 Dec 2023
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark Eldar Kurtic Torsten Hoefler Dan Alistarh 37 3 0 21 Dec 2023
Critical Influence of Overparameterization on Sharpness-aware Minimization Sungbin Shin Dongyeop Lee Maksym Andriushchenko Namhoon Lee AAML 47 1 0 29 Nov 2023