Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

31 January 2021

Dan Alistarh

Papers citing "Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks"

50 / 361 papers shown

Title
Low-Depth Spatial Tree Algorithms Yves Baumann Tal Ben-Nun Maciej Besta Lukas Gianinazzi Torsten Hoefler Piotr Luczynski 42 0 0 19 Apr 2024
Parallel Decoding via Hidden Transfer for Lossless Large Language Model Acceleration Pengfei Wu Jiahao Liu Zhuocheng Gong Qifan Wang Jinpeng Li Jingang Wang Xunliang Cai Dongyan Zhao 28 1 0 18 Apr 2024
Decoupled Weight Decay for Any $p$ Norm N. Outmezguine Noam Levi 27 2 0 16 Apr 2024
SparseDM: Toward Sparse Efficient Diffusion Models Kafeng Wang Jianfei Chen He Li Zhenpeng Mi Jun-Jie Zhu DiffM 70 8 0 16 Apr 2024
CATS: Contextually-Aware Thresholding for Sparsity in Large Language Models Je-Yong Lee Donghyun Lee Genghan Zhang Mo Tiwari Azalia Mirhoseini 44 15 0 12 Apr 2024
Learning smooth functions in high dimensions: from sparse polynomials to deep neural networks Ben Adcock Simone Brugiapaglia N. Dexter S. Moraga 42 4 0 04 Apr 2024
Talaria: Interactively Optimizing Machine Learning Models for Efficient Inference Fred Hohman Chaoqun Wang Jinmook Lee Jochen Görtler Dominik Moritz Jeffrey P. Bigham Zhile Ren Cecile Foret Qi Shan Xiaoyi Zhang 35 7 0 03 Apr 2024
Diagnosis of Skin Cancer Using VGG16 and VGG19 Based Transfer Learning Models Amir Faghihi Mohammadreza Fathollahi Roozbeh Rajabi 25 27 0 01 Apr 2024
Systematic construction of continuous-time neural networks for linear dynamical systems Chinmay Datar Adwait Datar Felix Dietrich W. Schilders AI4TS 51 1 0 24 Mar 2024
AI and Memory Wall A. Gholami Z. Yao Sehoon Kim Coleman Hooper Michael W. Mahoney Kurt Keutzer 27 146 0 21 Mar 2024
Searching Search Spaces: Meta-evolving a Geometric Encoding for Neural Networks Tarek Kunze Paul Templier Dennis G. Wilson 35 0 0 20 Mar 2024
Unifews: Unified Entry-Wise Sparsification for Efficient Graph Neural Network Ningyi Liao Zihao Yu Siqiang Luo GNN 35 0 0 20 Mar 2024
Adversarial Fine-tuning of Compressed Neural Networks for Joint Improvement of Robustness and Efficiency Hallgrimur Thorsteinsson Valdemar J Henriksen Tong Chen Raghavendra Selvan AAML 48 1 0 14 Mar 2024
FlexNN: A Dataflow-aware Flexible Deep Learning Accelerator for Energy-Efficient Edge Devices Arnab Raha Deepak A. Mathaikutty Soumendu Kumar Ghosh Shamik Kundu 24 7 0 14 Mar 2024
Random Search as a Baseline for Sparse Neural Network Architecture Search Rezsa Farahani 36 0 0 13 Mar 2024
Hunting Attributes: Context Prototype-Aware Learning for Weakly Supervised Semantic Segmentation Feilong Tang Zhongxing Xu Zhaojun Qu Wei Feng Xingjian Jiang Zongyuan Ge 61 7 0 12 Mar 2024
Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN Biswadeep Chakraborty Beomseok Kang H. Kumar Saibal Mukhopadhyay 46 8 0 06 Mar 2024
Not All Layers of LLMs Are Necessary During Inference Siqi Fan Xin Jiang Xiang Li Xuying Meng Peng Han Shuo Shang Aixin Sun Yequan Wang Zhongyuan Wang 51 33 0 04 Mar 2024
"Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach Lingyu Gu Yongqiang Du Yuan Zhang Di Xie Shiliang Pu Robert C. Qiu Zhenyu Liao 44 6 0 01 Mar 2024
Out-of-Distribution Detection using Neural Activation Prior Weilin Wan Weizhong Zhang Quan Zhou Fan Yi Cheng Jin OODD 40 0 0 28 Feb 2024
SparseLLM: Towards Global Pruning for Pre-trained Language Models Guangji Bai Yijiang Li Chen Ling Kibaek Kim Liang Zhao 33 7 0 28 Feb 2024
Efficient Backpropagation with Variance-Controlled Adaptive Sampling Ziteng Wang Jianfei Chen Jun Zhu BDL 45 2 0 27 Feb 2024
MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT Omkar Thawakar Ashmal Vayani Salman Khan Hisham Cholakal Rao M. Anwer M. Felsberg Timothy Baldwin Eric P. Xing Fahad Shahbaz Khan 54 31 0 26 Feb 2024
Deep Neural Network Initialization with Sparsity Inducing Activations Ilan Price Nicholas Daultry Ball Samuel C.H. Lam Adam C. Jones Jared Tanner AI4CE 31 1 0 25 Feb 2024
Shaving Weights with Occam's Razor: Bayesian Sparsification for Neural Networks Using the Marginal Likelihood Rayen Dhahri Alexander Immer Bertrand Charpentier Stephan Günnemann Vincent Fortuin BDL UQCV 35 4 0 25 Feb 2024
Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale Dan Zhao S. Samsi Joseph McDonald Baolin Li David Bestor Michael Jones Devesh Tiwari V. Gadepally 53 17 0 25 Feb 2024
ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity within Large Language Models Chenyang Song Xu Han Zhengyan Zhang Shengding Hu Xiyu Shi ... Chen Chen Zhiyuan Liu Guanglin Li Tao Yang Maosong Sun 53 25 0 21 Feb 2024
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding Zhuoming Chen Avner May Ruslan Svirschevski Yuhsun Huang Max Ryabinin Zhihao Jia Beidi Chen 48 41 0 19 Feb 2024
Neuron-centric Hebbian Learning Andrea Ferigo Elia Cunegatti Giovanni Iacca 36 0 0 16 Feb 2024
NutePrune: Efficient Progressive Pruning with Numerous Teachers for Large Language Models Shengrui Li Junzhe Chen Xueting Han Jing Bai 24 6 0 15 Feb 2024
Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers Abhimanyu Bambhaniya Amir Yazdanbakhsh Suvinay Subramanian Sheng-Chun Kao Shivani Agrawal Utku Evci Tushar Krishna 54 14 0 07 Feb 2024
Discovering interpretable models of scientific image data with deep learning Christopher J. Soelistyo Alan R. Lowe 13 6 0 05 Feb 2024
Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitate Can Jin Tong Che Hongwu Peng Yiyuan Li Dimitris N. Metaxas Marco Pavone 44 44 0 05 Feb 2024
Optimal Parameter and Neuron Pruning for Out-of-Distribution Detection Chao Chen Zhihang Fu Kai-Chun Liu Ze Chen Mingyuan Tao Jieping Ye OODD 38 3 0 04 Feb 2024
A Comprehensive Survey of Compression Algorithms for Language Models Seungcheol Park Jaehyeon Choi Sojin Lee U. Kang MQ 32 12 0 27 Jan 2024
SliceGPT: Compress Large Language Models by Deleting Rows and Columns Saleh Ashkboos Maximilian L. Croci Marcelo Gennari do Nascimento Torsten Hoefler James Hensman VLM 132 148 0 26 Jan 2024
Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference Jinghan Yao Quentin G. Anthony Hari Subramoni Hari Subramoni Dhabaleswar K. Panda MoE 39 13 0 16 Jan 2024
Edge-Enabled Anomaly Detection and Information Completion for Social Network Knowledge Graphs Fan Lu Quan Qi Huaibin Qin 8 2 0 13 Jan 2024
Always-Sparse Training by Growing Connections with Guided Stochastic Exploration Mike Heddes Narayan Srinivasa T. Givargis Alexandru Nicolau 91 0 0 12 Jan 2024
EsaCL: Efficient Continual Learning of Sparse Models Weijieying Ren V. Honavar CLL 33 3 0 11 Jan 2024
RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptation Mahdi Nikdan Soroush Tabesh Elvir Crnčević Dan Alistarh 16 27 0 09 Jan 2024
Robust Neural Pruning with Gradient Sampling Optimization for Residual Neural Networks Juyoung Yun 24 1 0 26 Dec 2023
PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs Max Zimmer Megi Andoni Christoph Spiegel Sebastian Pokutta VLM 55 10 0 23 Dec 2023
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction Pratyusha Sharma Jordan T. Ash Dipendra Kumar Misra LRM 19 79 0 21 Dec 2023
How to Prune Your Language Model: Recovering Accuracy on the "Sparsity May Cry'' Benchmark Eldar Kurtic Torsten Hoefler Dan Alistarh 42 3 0 21 Dec 2023
Adaptive Computation Modules: Granular Conditional Computation For Efficient Inference Bartosz Wójcik Alessio Devoto Karol Pustelnik Pasquale Minervini Simone Scardapane 27 5 0 15 Dec 2023
Language Modeling on a SpiNNaker 2 Neuromorphic Chip Khaleelulla Khan Nazeer Mark Schöne Rishav Mukherji Bernhard Vogginger Christian Mayr David Kappel Anand Subramoney 37 5 0 14 Dec 2023
An Incremental Unified Framework for Small Defect Inspection Jiaqi Tang Hao Lu Xiaogang Xu Ruizheng Wu Sixing Hu Tong Zhang Tsz Wa Cheng Ming Ge Ying-Cong Chen Fugee Tsung 31 5 0 14 Dec 2023
ELSA: Partial Weight Freezing for Overhead-Free Sparse Network Deployment Paniz Halvachi Alexandra Peste Dan Alistarh Christoph H. Lampert 33 0 0 11 Dec 2023
Exploring Sparsity in Graph Transformers Chuang Liu Yibing Zhan Xueqi Ma Liang Ding Dapeng Tao Jia Wu Wenbin Hu Bo Du 42 6 0 09 Dec 2023