Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

31 January 2021

Dan Alistarh

Papers citing "Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks"

50 / 361 papers shown

Title
JUNO: Optimizing High-Dimensional Approximate Nearest Neighbour Search with Sparsity-Aware Algorithm and Ray-Tracing Core Mapping Zihan Liu Wentao Ni Jingwen Leng Yu Feng Cong Guo Quan Chen Chao Li Minyi Guo Yuhao Zhu 27 14 0 04 Dec 2023
LinguaLinked: A Distributed Large Language Model Inference System for Mobile Devices Junchen Zhao Yurun Song Simeng Liu Ian G. Harris Sangeetha Abdu Jyothi 26 5 0 01 Dec 2023
Critical Influence of Overparameterization on Sharpness-aware Minimization Sungbin Shin Dongyeop Lee Maksym Andriushchenko Namhoon Lee AAML 50 1 0 29 Nov 2023
REDS: Resource-Efficient Deep Subnetworks for Dynamic Resource Constraints Francesco Corti Balz Maag Joachim Schauer U. Pferschy O. Saukh 49 2 0 22 Nov 2023
Efficient Neural Networks for Tiny Machine Learning: A Comprehensive Review M. Lê Pierre Wolinski Julyan Arbel 39 9 0 20 Nov 2023
Robustness Enhancement in Neural Networks with Alpha-Stable Training Noise Xueqiong Yuan Jipeng Li E. Kuruoglu OOD 13 3 0 17 Nov 2023
Polynomially Over-Parameterized Convolutional Neural Networks Contain Structured Strong Winning Lottery Tickets A. D. Cunha Francesco d’Amore Emanuele Natale MLT 27 1 0 16 Nov 2023
Activity Sparsity Complements Weight Sparsity for Efficient RNN Inference Rishav Mukherji Mark Schöne Khaleelulla Khan Nazeer Christian Mayr Anand Subramoney 38 2 0 13 Nov 2023
Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models Jan Finkbeiner Thomas Gmeinder M. Pupilli A. Titterton Emre Neftci 31 3 0 07 Nov 2023
MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters Chau Pham Piotr Teterwak Soren Nelson Bryan A. Plummer 17 3 0 07 Nov 2023
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time Zichang Liu Jue Wang Tri Dao Dinesh Manocha Binhang Yuan ... Anshumali Shrivastava Ce Zhang Yuandong Tian Christopher Ré Beidi Chen BDL 35 194 0 26 Oct 2023
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models Elias Frantar Dan Alistarh MQ MoE 34 25 0 25 Oct 2023
Federated learning compression designed for lightweight communications Lucas Grativol Ribeiro Mathieu Léonardon Guillaume Muller Virginie Fresse Matthieu Arzel FedML 32 3 0 23 Oct 2023
Breaking through Deterministic Barriers: Randomized Pruning Mask Generation and Selection Jianwei Li Weizhi Gao Qi Lei Dongkuan Xu 32 2 0 19 Oct 2023
Sparse Fine-tuning for Inference Acceleration of Large Language Models Eldar Kurtic Denis Kuznedelev Elias Frantar Michael Goin Dan Alistarh 35 8 0 10 Oct 2023
Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences Fred Hohman Mary Beth Kery Donghao Ren Dominik Moritz 37 16 0 06 Oct 2023
Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversion Filip Szatkowski Eric Elmoznino Younesse Kaddar Simone Scardapane MoE 41 5 0 06 Oct 2023
Design Principles for Lifelong Learning AI Accelerators Dhireesha Kudithipudi Anurag Daram Abdullah M. Zyarah Fatima Tuz Zohora J. Aimone ... Emre Neftci M. Mattina Vincenzo Lomonaco Clare D. Thiem Benjamin Epstein 53 17 0 05 Oct 2023
Extreme sparsification of physics-augmented neural networks for interpretable model discovery in mechanics J. Fuhg Reese E. Jones N. Bouklas AI4CE 34 23 0 05 Oct 2023
Sparse Deep Learning for Time Series Data: Theory and Applications Mingxuan Zhang Y. Sun Faming Liang AI4TS OOD BDL 44 2 0 05 Oct 2023
Feather: An Elegant Solution to Effective DNN Sparsification Athanasios Glentis Georgoulakis George Retsinas Petros Maragos 32 0 0 03 Oct 2023
VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores Roberto L. Castro Andrei Ivanov Diego Andrade Tal Ben-Nun B. Fraguela Torsten Hoefler 27 15 0 03 Oct 2023
Selective Feature Adapter for Dense Vision Transformers XueQing Deng Qi Fan Xiaojie Jin Linjie Yang Peng Wang 37 0 0 03 Oct 2023
The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks Cameron Shinn Collin McCarthy Saurav Muralidharan Muhammad Osama John Douglas Owens 18 2 0 30 Sep 2023
Smooth Exact Gradient Descent Learning in Spiking Neural Networks Christian Klos Raoul-Martin Memmesheimer 43 6 0 25 Sep 2023
Detach-ROCKET: Sequential feature selection for time series classification with random convolutional kernels Gonzalo Uribarri Federico Barone A. Ansuini Erik Fransén AI4TS 50 6 0 25 Sep 2023
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity Haojun Xia Zhen Zheng Yuchao Li Donglin Zhuang Zhongzhu Zhou Xiafei Qiu Yong Li Wei Lin Shuaiwen Leon Song 67 11 0 19 Sep 2023
Scaling Laws for Sparsely-Connected Foundation Models Elias Frantar C. Riquelme N. Houlsby Dan Alistarh Utku Evci 35 36 0 15 Sep 2023
Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity Matteo Grimaldi Darshan C. Ganji Ivan Lazarevich Sudhakar Sah 14 10 0 12 Sep 2023
QuantEase: Optimization-based Quantization for Language Models Kayhan Behdin Ayan Acharya Aman Gupta Qingquan Song Siyu Zhu S. Keerthi Rahul Mazumder MQ 30 20 0 05 Sep 2023
Equitable-FL: Federated Learning with Sparsity for Resource-Constrained Environment Indrajeet Kumar Sinha Shekhar Verma Krishna Pratap Singh FedML 40 0 0 02 Sep 2023
Continual Learning with Dynamic Sparse Training: Exploring Algorithms for Effective Model Updates Murat Onur Yildirim Elif Ceren Gok Yildirim Ghada Sokar Decebal Constantin Mocanu Joaquin Vanschoren CLL 30 7 0 28 Aug 2023
Homological Convolutional Neural Networks Antonio Briola Yuanrong Wang Silvia Bartolucci T. Aste LMTD 33 6 0 26 Aug 2023
A2Q: Accumulator-Aware Quantization with Guaranteed Overflow Avoidance Ian Colbert Alessandro Pappalardo Jakoba Petri-Koenig MQ 24 9 0 25 Aug 2023
Multi-Objective Optimization for Sparse Deep Multi-Task Learning S. S. Hotegni M. Berkemeier S. Peitz 25 6 0 23 Aug 2023
Less is More -- Towards parsimonious multi-task models using structured sparsity Richa Upadhyay Ronald Phlypo Rajkumar Saini Marcus Liwicki MoE 25 3 0 23 Aug 2023
HyperSparse Neural Networks: Shifting Exploration to Exploitation through Adaptive Regularization Patrick Glandorf Timo Kaiser Bodo Rosenhahn 49 5 0 14 Aug 2023
Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modeling Marija vSakota Maxime Peyrard Robert West 32 46 0 11 Aug 2023
FPGA Resource-aware Structured Pruning for Real-Time Neural Networks Benjamin Ramhorst Vladimir Loncar George A. Constantinides 33 4 0 09 Aug 2023
Attention-Driven Lightweight Model for Pigmented Skin Lesion Detection Mingzhe Hu Xiaofeng Yang MedIm 28 0 0 04 Aug 2023
Accurate Neural Network Pruning Requires Rethinking Sparse Optimization Denis Kuznedelev Eldar Kurtic Eugenia Iofinova Elias Frantar Alexandra Peste Dan Alistarh VLM 35 11 0 03 Aug 2023
Unlocking the Emotional World of Visual Media: An Overview of the Science, Research, and Impact of Understanding Emotion James Z. Wang Sicheng Zhao Chenyan Wu Reginald B. Adams M. Newman T. Shafir Rachelle Tsachor 38 30 0 25 Jul 2023
Model Compression Methods for YOLOv5: A Review Mohammad Jani Jamil Fayyad Younes Al Younes Homayoun Najjaran 36 14 0 21 Jul 2023
Differentiable Transportation Pruning Yun-qiang Li Jan van Gemert Torsten Hoefler Bert Moons E. Eleftheriou Bram-Ernst Verhoef OT 24 7 0 17 Jul 2023
Sparse Model Soups: A Recipe for Improved Pruning via Model Averaging Max Zimmer Christoph Spiegel Sebastian Pokutta MoMe 49 14 0 29 Jun 2023
H $_2$ O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models Zhenyu Zhang Ying Sheng Dinesh Manocha Tianlong Chen Lianmin Zheng ... Yuandong Tian Christopher Ré Clark W. Barrett Zhangyang Wang Beidi Chen VLM 66 261 0 24 Jun 2023
Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training A. Nowak Bram Grooten Decebal Constantin Mocanu Jacek Tabor 33 9 0 21 Jun 2023
Quantifying lottery tickets under label noise: accuracy, calibration, and complexity V. Arora Daniele Irto Sebastian Goldt G. Sanguinetti 52 2 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 90 361 0 20 Jun 2023
Gradient is All You Need? Konstantin Riedl T. Klock Carina Geldhauser M. Fornasier 35 6 0 16 Jun 2023