Sharing Attention Weights for Fast Transformer

26 June 2019

Jingbo Zhu

Papers citing "Sharing Attention Weights for Fast Transformer"

21 / 21 papers shown

Title
Tensor Product Attention Is All You Need Yifan Zhang Yifeng Liu Huizhuo Yuan Zhen Qin Yang Yuan Q. Gu Andrew Chi-Chih Yao 96 9 0 11 Jan 2025
PSformer: Parameter-efficient Transformer with Segment Attention for Time Series Forecasting Yanlong Wang Jinfeng Xu Fei Ma Shao-Lun Huang Danny Dongning Sun Xiao-Ping Zhang AI4TS 45 1 0 03 Nov 2024
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 30 3 0 07 Aug 2023
Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation Kangwook Jang Sungnyun Kim Se-Young Yun Hoi-Rim Kim 34 5 0 19 May 2023
Transformers in Speech Processing: A Survey S. Latif Aun Zaidi Heriberto Cuayáhuitl Fahad Shamshad Moazzam Shoukat Junaid Qadir 46 47 0 21 Mar 2023
Exploring Attention Map Reuse for Efficient Transformer Neural Networks Kyuhong Shim Jungwook Choi Wonyong Sung ViT 26 3 0 29 Jan 2023
Skip-Attention: Improving Vision Transformers by Paying Less Attention Shashanka Venkataramanan Amir Ghodrati Yuki M. Asano Fatih Porikli A. Habibian ViT 23 25 0 05 Jan 2023
ACORT: A Compact Object Relation Transformer for Parameter Efficient Image Captioning J. Tan Y. Tan C. Chan Joon Huang Chuah VLM ViT 31 15 0 11 Feb 2022
Bag of Tricks for Optimizing Transformer Efficiency Ye Lin Yanyang Li Tong Xiao Jingbo Zhu 34 6 0 09 Sep 2021
LazyFormer: Self Attention with Lazy Update Chengxuan Ying Guolin Ke Di He Tie-Yan Liu 25 15 0 25 Feb 2021
An Efficient Transformer Decoder with Compressed Sub-layers Yanyang Li Ye Lin Tong Xiao Jingbo Zhu 33 29 0 03 Jan 2021
Learning Light-Weight Translation Models from Deep Transformer Bei Li Ziyang Wang Hui Liu Quan Du Tong Xiao Chunliang Zhang Jingbo Zhu VLM 120 40 0 27 Dec 2020
Exploiting Neural Query Translation into Cross Lingual Information Retrieval Liang Yao Baosong Yang Haibo Zhang Weihua Luo Boxing Chen 22 12 0 26 Oct 2020
Rethinking Attention with Performers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Afroz Mohiuddin Lukasz Kaiser David Belanger Lucy J. Colwell Adrian Weller 71 1,527 0 30 Sep 2020
Towards Fully 8-bit Integer Inference for the Transformer Model Ye Lin Yanyang Li Tengbo Liu Tong Xiao Tongran Liu Jingbo Zhu MQ 11 62 0 17 Sep 2020
An Overview of Neural Network Compression James OÑeill AI4CE 47 98 0 05 Jun 2020
Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers K. Choromanski Valerii Likhosherstov David Dohan Xingyou Song Andreea Gane ... Peter Hawkins Jared Davis David Belanger Lucy J. Colwell Adrian Weller 39 84 0 05 Jun 2020
Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation Po-Han Chi Pei-Hung Chung Tsung-Han Wu Chun-Cheng Hsieh Yen-Hao Chen Shang-Wen Li Hung-yi Lee SSL 9 147 0 18 May 2020
The Cascade Transformer: an Application for Efficient Answer Sentence Selection Luca Soldaini Alessandro Moschitti 29 44 0 05 May 2020
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Zhehuai Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 718 6,750 0 26 Sep 2016
Effective Approaches to Attention-based Neural Machine Translation Thang Luong Hieu H. Pham Christopher D. Manning 220 7,930 0 17 Aug 2015