Achieving Sparse Activation in Small Language Models

3 June 2024

Papers citing "Achieving Sparse Activation in Small Language Models"

10 / 10 papers shown

Title
Small Language Models: Survey, Measurements, and Insights Zhenyan Lu Xiang Li Dongqi Cai Rongjie Yi Fangming Liu Xiwen Zhang Nicholas D. Lane Mengwei Xu ObjD LRM 87 44 0 24 Sep 2024
Memory-Efficient Fine-Tuning of Compressed Large Language Models via sub-4-bit Integer Quantization Jeonghoon Kim J. H. Lee Sungdong Kim Joonsuk Park Kang Min Yoo S. Kwon Dongsoo Lee MQ 71 102 0 23 May 2023
Fast Inference from Transformers via Speculative Decoding Yaniv Leviathan Matan Kalman Yossi Matias LRM 67 663 0 30 Nov 2022
Attribution-based XAI Methods in Computer Vision: A Review Kumar Abhishek Deeksha Kamath 42 18 0 27 Nov 2022
SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance Edouard Yvinec Arnaud Dapogny Matthieu Cord Kévin Bailly 54 9 0 08 Jul 2022
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 53 121 0 05 Oct 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 230 703 0 31 Jan 2021
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 59 1,049 0 25 May 2019
SNIP: Single-shot Network Pruning based on Connection Sensitivity Namhoon Lee Thalaiyasingam Ajanthan Philip Torr VLM 187 1,190 0 04 Oct 2018
Axiomatic Attribution for Deep Networks Mukund Sundararajan Ankur Taly Qiqi Yan OOD FAtt 79 5,920 0 04 Mar 2017