Rethinking the Value of Transformer Components

7 November 2020

Papers citing "Rethinking the Value of Transformer Components"

13 / 13 papers shown

Title
Outlier Weighed Layerwise Sparsity (OWL): A Missing Secret Sauce for Pruning LLMs to High Sparsity Lu Yin You Wu Zhenyu (Allen) Zhang Cheng-Yu Hsieh Yaqing Wang ... Mykola Pechenizkiy Yi Liang Michael Bendersky Zhangyang Wang Shiwei Liu 28 79 0 08 Oct 2023
Transformer-based models and hardware acceleration analysis in autonomous driving: A survey J. Zhong Zheng Liu Xiangshan Chen ViT 44 17 0 21 Apr 2023
Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularization Jianping Zhang Yizhan Huang Weibin Wu Michael R. Lyu AAML ViT 18 50 0 28 Mar 2023
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 72 528 0 13 Jun 2022
Understanding and Mitigating the Uncertainty in Zero-Shot Translation Wenxuan Wang Wenxiang Jiao Shuo Wang Zhaopeng Tu Michael R. Lyu UQLM 35 9 0 20 May 2022
Training-free Transformer Architecture Search Qinqin Zhou Kekai Sheng Xiawu Zheng Ke Li Xing Sun Yonghong Tian Jie Chen Rongrong Ji ViT 40 46 0 23 Mar 2022
Kformer: Knowledge Injection in Transformer Feed-Forward Layers Yunzhi Yao Shaohan Huang Li Dong Furu Wei Huajun Chen Ningyu Zhang KELM MedIm 29 42 0 15 Jan 2022
MoEfication: Transformer Feed-forward Layers are Mixtures of Experts Zhengyan Zhang Yankai Lin Zhiyuan Liu Peng Li Maosong Sun Jie Zhou MoE 27 117 0 05 Oct 2021
Towards Efficient Post-training Quantization of Pre-trained Language Models Haoli Bai Lu Hou Lifeng Shang Xin Jiang Irwin King M. Lyu MQ 82 47 0 30 Sep 2021
Bag of Tricks for Optimizing Transformer Efficiency Ye Lin Yanyang Li Tong Xiao Jingbo Zhu 34 6 0 09 Sep 2021
How Does Selective Mechanism Improve Self-Attention Networks? Xinwei Geng Longyue Wang Xing Wang Bing Qin Ting Liu Zhaopeng Tu AAML 39 35 0 03 May 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 224 383 0 05 Mar 2020
Improving neural networks by preventing co-adaptation of feature detectors Geoffrey E. Hinton Nitish Srivastava A. Krizhevsky Ilya Sutskever Ruslan Salakhutdinov VLM 266 7,638 0 03 Jul 2012