Weight Sparsity Complements Activity Sparsity in Neuromorphic Language
Models

Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models

1 May 2024

Rishav Mukherji

Khaleelulla Khan Nazeer

Anand Subramoney

Papers citing "Weight Sparsity Complements Activity Sparsity in Neuromorphic Language Models"

19 / 19 papers shown

Title
Sparse Spiking Neural Network: Exploiting Heterogeneity in Timescales for Pruning Recurrent SNN Biswadeep Chakraborty Beomseok Kang H. Kumar Saibal Mukhopadhyay 99 11 0 06 Mar 2024
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 205 593 0 22 May 2023
NeuroBench: A Framework for Benchmarking Neuromorphic Computing Algorithms and Systems Jason Yik Korneel Van den Berghe Douwe den Blanken Younes Bouhadjar Maxime Fabre ... Fatima Tuz Zohora Charlotte Frenkel Vijay Janapa Reddi Charlotte Frenkel Vijay Janapa Reddi 82 21 0 10 Apr 2023
Exploring Lottery Ticket Hypothesis in Spiking Neural Networks Youngeun Kim Yuhang Li Hyoungseob Park Yeshwanth Venkatesha Ruokai Yin Priyadarshini Panda 102 48 0 04 Jul 2022
Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks Kevin Lee Hunter Lawrence Spracklen Subutai Ahmad 60 20 0 27 Dec 2021
Spartus: A 9.4 TOp/s FPGA-based LSTM Accelerator Exploiting Spatio-Temporal Sparsity Chang Gao T. Delbruck Shih-Chii Liu 56 45 0 04 Aug 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 311 718 0 31 Jan 2021
Connection Pruning for Deep Spiking Neural Networks with On-Chip Learning Thao N. N. Nguyen B. Veeravalli Xuanyao Fong 48 14 0 09 Oct 2020
Long Short-Term Memory Spiking Networks and Their Applications Ali Lotfi-Rezaabad S. Vishwanath 47 62 0 09 Jul 2020
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model Zhilin Yang Zihang Dai Ruslan Salakhutdinov William W. Cohen BDL 63 371 0 10 Nov 2017
STDP Based Pruning of Connections and Weight Quantization in Spiking Neural Networks for Energy Efficient Recognition Nitin Rathi Priyadarshini Panda Kaushik Roy 47 113 0 12 Oct 2017
To prune, or not to prune: exploring the efficacy of pruning for model compression Michael Zhu Suyog Gupta 191 1,276 0 05 Oct 2017
Exploring Sparsity in Recurrent Neural Networks Sharan Narang Erich Elsen G. Diamos Shubho Sengupta 38 312 0 17 Apr 2017
Delta Networks for Optimized Recurrent Network Computation Daniel Neil Junhaeng Lee T. Delbruck Shih-Chii Liu 83 66 0 16 Dec 2016
Improving Neural Language Models with a Continuous Cache Edouard Grave Armand Joulin Nicolas Usunier KELM 48 300 0 13 Dec 2016
Sigma Delta Quantized Networks Peter O'Connor Max Welling 52 48 0 07 Nov 2016
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 308 2,854 0 26 Sep 2016
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 310 6,672 0 08 Jun 2015
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation Kyunghyun Cho B. V. Merrienboer Çağlar Gülçehre Dzmitry Bahdanau Fethi Bougares Holger Schwenk Yoshua Bengio AIMat 1.0K 23,344 0 03 Jun 2014