Leveraging redundancy in attention with Reuse Transformers

13 October 2021

Srinadh Bhojanapalli

Sanjiv Kumar

Papers citing "Leveraging redundancy in attention with Reuse Transformers"

20 / 20 papers shown

Title
MSPLoRA: A Multi-Scale Pyramid Low-Rank Adaptation for Efficient Model Fine-Tuning Jiancheng Zhao Xingda Yu Zhen Yang MoE 57 1 0 27 Mar 2025
AlphaPruning: Using Heavy-Tailed Self Regularization Theory for Improved Layer-wise Pruning of Large Language Models Haiquan Lu Yefan Zhou Shiwei Liu Zhangyang Wang Michael W. Mahoney Yaoqing Yang 29 0 0 14 Oct 2024
Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression Jingcun Wang Yu-Guang Chen Ing-Chao Lin Bing Li Grace Li Zhang 35 4 0 02 Oct 2024
EchoAtt: Attend, Copy, then Adjust for More Efficient Large Language Models Hossein Rajabzadeh A. Jafari Aman Sharma Benyamin Jami Hyock Ju Kwon Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 32 0 0 22 Sep 2024
EfficientASR: Speech Recognition Network Compression via Attention Redundancy and Chunk-Level FFN Optimization Jianzong Wang Ziqi Liang Xulong Zhang Ning Cheng Jing Xiao 38 0 0 30 Apr 2024
MLP Can Be A Good Transformer Learner Sihao Lin Pumeng Lyu Dongrui Liu Tao Tang Xiaodan Liang Andy Song Xiaojun Chang ViT 39 11 0 08 Apr 2024
Understanding Neural Network Binarization with Forward and Backward Proximal Quantizers Yiwei Lu Yaoliang Yu Xinlin Li Vahid Partovi Nia MQ 38 3 0 27 Feb 2024
MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT Omkar Thawakar Ashmal Vayani Salman Khan Hisham Cholakal Rao M. Anwer M. Felsberg Timothy Baldwin Eric P. Xing Fahad Shahbaz Khan 46 31 0 26 Feb 2024
Head-wise Shareable Attention for Large Language Models Zouying Cao Yifei Yang Hai Zhao 41 4 0 19 Feb 2024
Dynamic Layer Tying for Parameter-Efficient Transformers Tamir David Hay Lior Wolf 27 3 0 23 Jan 2024
Fast Sampling Through The Reuse Of Attention Maps In Diffusion Models Rosco Hunter L. Dudziak Mohamed S. Abdelfattah Abhinav Mehrotra Sourav Bhattacharya Hongkai Wen 23 1 0 13 Dec 2023
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 78 0 08 Oct 2023
Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation Kangwook Jang Sungnyun Kim Se-Young Yun Hoi-Rim Kim 29 5 0 19 May 2023
AttMEMO : Accelerating Transformers with Memoization on Big Memory Systems Yuan Feng Hyeran Jeon F. Blagojevic Cyril Guyot Qing Li Dong Li GNN 24 3 0 23 Jan 2023
Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition Ruiqi Wang Wonse Jo Dezhong Zhao Weizheng Wang B. Yang Guohua Chen Byung-Cheol Min HAI 26 28 0 30 Sep 2022
On The Computational Complexity of Self-Attention Feyza Duman Keles Pruthuvi Maheshakya Wijewardena C. Hegde 70 108 0 11 Sep 2022
Visualizing and Understanding Patch Interactions in Vision Transformer Jie Ma Yalong Bai Bineng Zhong Wei Zhang Ting Yao Tao Mei ViT 23 32 0 11 Mar 2022
Q-ViT: Fully Differentiable Quantization for Vision Transformer Zhexin Li Tong Yang Peisong Wang Jian Cheng ViT MQ 33 41 0 19 Jan 2022
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,015 0 28 Jul 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018