Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

31 January 2021

Dan Alistarh

Papers citing "Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks"

50 / 361 papers shown

Title
Spatial Re-parameterization for N:M Sparsity Yuxin Zhang Mingbao Lin Mingliang Xu Yonghong Tian Rongrong Ji 46 2 0 09 Jun 2023
Recovering Simultaneously Structured Data via Non-Convex Iteratively Reweighted Least Squares C. Kümmerle J. Maly 25 1 0 08 Jun 2023
Randomized Schur Complement Views for Graph Contrastive Learning Vignesh Kothapalli 21 1 0 06 Jun 2023
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression Tim Dettmers Ruslan Svirschevski Vage Egiazarian Denis Kuznedelev Elias Frantar Saleh Ashkboos Alexander Borzunov Torsten Hoefler Dan Alistarh MQ 37 231 0 05 Jun 2023
TIES-Merging: Resolving Interference When Merging Models Prateek Yadav Derek Tam Leshem Choshen Colin Raffel Joey Tianyi Zhou MoMe 65 261 0 02 Jun 2023
Learning Task-preferred Inference Routes for Gradient De-conflict in Multi-output DNNs Yi Sun Xin Xu Jiaqiang Li Xiaochang Hu Yifei Shi L. Zeng 37 2 0 31 May 2023
Neural Network Reduction with Guided Regularizers Ali Haisam Muhammad Rafid Adrian Sandu 82 1 0 29 May 2023
Adaptive Sparsity Level during Training for Efficient Time Series Forecasting with Transformers Zahra Atashgahi Mykola Pechenizkiy Raymond N. J. Veldhuis Decebal Constantin Mocanu AI4TS AI4CE 34 1 0 28 May 2023
HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity Yannan Nellie Wu Po-An Tsai Saurav Muralidharan A. Parashar Vivienne Sze J. Emer 37 23 0 22 May 2023
Probabilistic Modeling: Proving the Lottery Ticket Hypothesis in Spiking Neural Network Man Yao Yu-Liang Chou Guangshe Zhao Xiawu Zheng Yonghong Tian Boxing Xu Guoqi Li 36 4 0 20 May 2023
Learning Activation Functions for Sparse Neural Networks Mohammad Loni Aditya Mohan Mehdi Asadi Marius Lindauer 27 4 0 18 May 2023
Functional Equivalence and Path Connectivity of Reducible Hyperbolic Tangent Networks Matthew Farrugia-Roberts 30 4 0 08 May 2023
Dynamic Sparse Training with Structured Sparsity Mike Lasby A. Golubeva Utku Evci Mihai Nica Yani Andrew Ioannou 36 19 0 03 May 2023
JaxPruner: A concise library for sparsity research Jooyoung Lee Wonpyo Park Nicole Mitchell Jonathan Pilault J. Obando-Ceron ... Hong-Seok Kim Yann N. Dauphin Karolina Dziugaite Pablo Samuel Castro Utku Evci 48 14 0 27 Apr 2023
Sparsified Model Zoo Twins: Investigating Populations of Sparsified Neural Network Models D. Honegger Konstantin Schurholt Damian Borth 37 4 0 26 Apr 2023
Towards Compute-Optimal Transfer Learning Massimo Caccia Alexandre Galashov Arthur Douillard Amal Rannen-Triki Dushyant Rao Michela Paganini Laurent Charlin MarcÁurelio Ranzato Razvan Pascanu 26 3 0 25 Apr 2023
Expand-and-Cluster: Parameter Recovery of Neural Networks Flavio Martinelli Berfin Simsek W. Gerstner Johanni Brea 32 5 0 25 Apr 2023
Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures Eugenia Iofinova Alexandra Peste Dan Alistarh 28 9 0 25 Apr 2023
Harnessing Deep Learning and HPC Kernels via High-Level Loop and Tensor Abstractions on CPU Architectures E. Georganas Dhiraj D. Kalamkar K. Voronin Abhisek Kundu Antonio Noack Hans Pabst Alexander Breuer A. Heinecke 16 2 0 25 Apr 2023
STen: Productive and Efficient Sparsity in PyTorch Andrei Ivanov Nikoli Dryden Tal Ben-Nun Saleh Ashkboos Torsten Hoefler 36 4 0 15 Apr 2023
Exploring Incompatible Knowledge Transfer in Few-shot Image Generation Yunqing Zhao Chao Du Milad Abdollahzadeh Tianyu Pang Min Lin Shuicheng Yan Ngai-man Cheung 38 19 0 15 Apr 2023
AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks Abhisek Kundu Naveen Mellempudi Dharma Teja Vooturi Bharat Kaul Pradeep Dubey 34 1 0 14 Apr 2023
Self-building Neural Networks Andrea Ferigo Giovanni Iacca 15 2 0 03 Apr 2023
PopSparse: Accelerated block sparse matrix multiplication on IPU Zhiyi Li Douglas Orr V. Ohan Godfrey Da Costa Tom Murray Adam Sanders D. Beker Dominic Masters 27 1 0 29 Mar 2023
CP-CNN: Core-Periphery Principle Guided Convolutional Neural Network Lin Zhao Haixing Dai Zihao Wu Dajiang Zhu Tianming Liu 38 1 0 27 Mar 2023
Vision Models Can Be Efficiently Specialized via Few-Shot Task-Aware Compression Denis Kuznedelev Soroush Tabesh Kimia Noorbakhsh Elias Frantar Sara Beery Eldar Kurtic Dan Alistarh MQ VLM 31 2 0 25 Mar 2023
Induced Feature Selection by Structured Pruning Nathan Hubens V. Delvigne M. Mancas B. Gosselin Marius Preda T. Zaharia 22 0 0 20 Mar 2023
SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models Vithursan Thangarasa Abhay Gupta William Marshall Tianda Li Kevin Leong D. DeCoste Sean Lie Shreyas Saxena MoE AI4CE 29 18 0 18 Mar 2023
Performance Embeddings: A Similarity-based Approach to Automatic Performance Optimization Lukas Trumper Tal Ben-Nun Philipp Schaad A. Calotoiu Torsten Hoefler 45 0 0 14 Mar 2023
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU Ying Sheng Lianmin Zheng Binhang Yuan Zhuohan Li Max Ryabinin ... Joseph E. Gonzalez Percy Liang Christopher Ré Ion Stoica Ce Zhang 149 372 0 13 Mar 2023
Supervised Feature Selection with Neuron Evolution in Sparse Neural Networks Zahra Atashgahi Xuhao Zhang Neil Kichler Shiwei Liu Lu Yin Mykola Pechenizkiy Raymond N. J. Veldhuis Decebal Constantin Mocanu 18 10 0 10 Mar 2023
Towards provably efficient quantum algorithms for large-scale machine-learning models Junyu Liu Minzhao Liu Jin-Peng Liu Ziyu Ye Yunfei Wang Yuri Alexeev Jens Eisert Liang Jiang 66 52 0 06 Mar 2023
Structured Pruning for Deep Convolutional Neural Networks: A survey Yang He Lingao Xiao 3DPC 30 122 0 01 Mar 2023
A Unified Framework for Soft Threshold Pruning Yanqing Chen Zhengyu Ma Wei Fang Xiawu Zheng Zhaofei Yu Yonghong Tian 88 19 0 25 Feb 2023
Neural-based classification rule learning for sequential data Marine Collery Philippe Bonnard Franccois Fages R. Kusters 17 4 0 22 Feb 2023
Considering Layerwise Importance in the Lottery Ticket Hypothesis Benjamin Vandersmissen José Oramas 37 1 0 22 Feb 2023
Multiobjective Evolutionary Pruning of Deep Neural Networks with Transfer Learning for improving their Performance and Robustness Javier Poyatos Daniel Molina Aitor Martínez Javier Del Ser Francisco Herrera 37 10 0 20 Feb 2023
TA-MoE: Topology-Aware Large Scale Mixture-of-Expert Training Chang-Qin Chen Min Li Zhihua Wu Dianhai Yu Chao Yang MoE 21 14 0 20 Feb 2023
Towards Optimal Compression: Joint Pruning and Quantization Ben Zandonati Glenn Bucagu Adrian Alan Pol M. Pierini Olya Sirkin Tal Kopetz MQ 25 2 0 15 Feb 2023
Automatic Noise Filtering with Dynamic Sparse Training in Deep Reinforcement Learning Bram Grooten Ghada Sokar Shibhansh Dohare Elena Mocanu Matthew E. Taylor Mykola Pechenizkiy Decebal Constantin Mocanu 27 11 0 13 Feb 2023
Bi-directional Masks for Efficient N:M Sparse Training Yuxin Zhang Yiting Luo Mingbao Lin Mingliang Xu Jingjing Xie Rongrong Ji Rongrong Ji 52 15 0 13 Feb 2023
Pruning Deep Neural Networks from a Sparsity Perspective Enmao Diao G. Wang Jiawei Zhan Yuhong Yang Jie Ding Vahid Tarokh 27 30 0 11 Feb 2023
SparseProp: Efficient Sparse Backpropagation for Faster Training of Neural Networks Mahdi Nikdan Tommaso Pegolotti Eugenia Iofinova Eldar Kurtic Dan Alistarh 26 11 0 09 Feb 2023
ZipLM: Inference-Aware Structured Pruning of Language Models Eldar Kurtic Elias Frantar Dan Alistarh MQ 25 24 0 07 Feb 2023
Ten Lessons We Have Learned in the New "Sparseland": A Short Handbook for Sparse Neural Network Researchers Shiwei Liu Zhangyang Wang 37 30 0 06 Feb 2023
Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance Ian Colbert Alessandro Pappalardo Jakoba Petri-Koenig MQ 27 4 0 31 Jan 2023
Equivariant Differentially Private Deep Learning: Why DP-SGD Needs Sparser Models Florian A. Hölzl Daniel Rueckert Georgios Kaissis 32 4 0 30 Jan 2023
A Stability Analysis of Fine-Tuning a Pre-Trained Model Z. Fu Anthony Man-Cho So Nigel Collier 23 3 0 24 Jan 2023
Unifying Synergies between Self-supervised Learning and Dynamic Computation Tarun Krishna Ayush K. Rai Alexandru Drimbarean Eric Arazo Paul Albert Alan F. Smeaton Kevin McGuinness Noel E. O'Connor 24 0 0 22 Jan 2023
Getting Away with More Network Pruning: From Sparsity to Geometry and Linear Regions Junyang Cai Khai-Nguyen Nguyen Nishant Shrestha Aidan Good Ruisen Tu Xin Yu Shandian Zhe Thiago Serra MLT 40 7 0 19 Jan 2023