Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 406 papers shown

Title
A Unified Pruning Framework for Vision Transformers Hao Yu Jianxin Wu ViT 61 63 0 30 Nov 2021
Can depth-adaptive BERT perform better on binary classification tasks Jing Fan Xin Zhang Sheng Zhang Yan Pan Lixiang Guo MQ 40 0 0 22 Nov 2021
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale Arun Babu Changhan Wang Andros Tjandra Kushal Lakhotia Qiantong Xu ... Yatharth Saraf J. Pino Alexei Baevski Alexis Conneau Michael Auli SSL 114 712 0 17 Nov 2021
A Survey on Green Deep Learning Jingjing Xu Wangchunshu Zhou Zhiyi Fu Hao Zhou Lei Li VLM 203 84 0 08 Nov 2021
Magic Pyramid: Accelerating Inference with Early Exiting and Token Pruning Xuanli He I. Keivanloo Yi Xu Xiang He Belinda Zeng Santosh Rajagopalan Trishul Chilimbi 81 19 0 30 Oct 2021
Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures Archit Parnami Rahul Singh Tarun Joshi 69 5 0 28 Oct 2021
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 296 1,911 0 26 Oct 2021
When in Doubt, Summon the Titans: Efficient Inference with Large Models A. S. Rawat Manzil Zaheer A. Menon Amr Ahmed Sanjiv Kumar 38 7 0 19 Oct 2021
Multilingual Domain Adaptation for NMT: Decoupling Language and Domain Information with Adapters Asa Cooper Stickland Alexandre Berard Vassilina Nikoulina AI4CE 67 29 0 18 Oct 2021
SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer Tu Vu Brian Lester Noah Constant Rami Al-Rfou Daniel Cer VLM LRM 223 290 0 15 Oct 2021
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline Xiangyang Liu Tianxiang Sun Junliang He Jiawen Wu Lingling Wu Xinyu Zhang Hao Jiang Bo Zhao Xuanjing Huang Xipeng Qiu ELM 85 47 0 13 Oct 2021
Word Order Does Not Matter For Speech Recognition Vineel Pratap Qiantong Xu Tatiana Likhomanenko Gabriel Synnaeve R. Collobert 75 4 0 12 Oct 2021
LightSeq2: Accelerated Training for Transformer-based Models on GPUs Xiaohui Wang Yang Wei Ying Xiong Guyue Huang Xian Qian Yufei Ding Mingxuan Wang Lei Li VLM 62 33 0 12 Oct 2021
SuperShaper: Task-Agnostic Super Pre-training of BERT Models with Variable Hidden Dimensions Vinod Ganesan Gowtham Ramesh Pratyush Kumar 63 9 0 10 Oct 2021
Exploring Heterogeneous Characteristics of Layers in ASR Models for More Efficient Training Lillian Zhou Dhruv Guliani Andreas Kabel Giovanni Motta F. Beaufays 68 1 0 08 Oct 2021
ABC: Attention with Bounded-memory Control Hao Peng Jungo Kasai Nikolaos Pappas Dani Yogatama Zhaofeng Wu Lingpeng Kong Roy Schwartz Noah A. Smith 125 22 0 06 Oct 2021
Towards Efficient Post-training Quantization of Pre-trained Language Models Haoli Bai Lu Hou Lifeng Shang Xin Jiang Irwin King Michael R. Lyu MQ 147 48 0 30 Sep 2021
Fine-tuning wav2vec2 for speaker recognition Nik Vaessen David A. van Leeuwen 116 108 0 30 Sep 2021
Long-Range Transformers for Dynamic Spatiotemporal Forecasting J. E. Grigsby Zhe Wang Nam Nguyen Yanjun Qi AI4TS 121 95 0 24 Sep 2021
RAIL-KD: RAndom Intermediate Layer Mapping for Knowledge Distillation Md. Akmal Haidar Nithin Anchuri Mehdi Rezagholizadeh Abbas Ghaddar Philippe Langlais Pascal Poupart 111 22 0 21 Sep 2021
Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognition Felix Wu Kwangyoun Kim Jing Pan Kyu Jeong Han Kilian Q. Weinberger Yoav Artzi 60 75 0 14 Sep 2021
Block Pruning For Faster Transformers François Lagunas Ella Charlaix Victor Sanh Alexander M. Rush VLM 74 225 0 10 Sep 2021
What's Hidden in a One-layer Randomly Weighted Transformer? Sheng Shen Z. Yao Douwe Kiela Kurt Keutzer Michael W. Mahoney 50 4 0 08 Sep 2021
Layer-wise Model Pruning based on Mutual Information Chun Fan Jiwei Li Xiang Ao Leilei Gan Yuxian Meng Xiaofei Sun 84 19 0 28 Aug 2021
Towards Structured Dynamic Sparse Pre-Training of BERT A. Dietrich Frithjof Gressmann Douglas Orr Ivan Chelombiev Daniel Justus Carlo Luschi 66 17 0 13 Aug 2021
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing Katikapalli Subramanyam Kalyan A. Rajasekharan S. Sangeetha VLM LM&MA 107 270 0 12 Aug 2021
On the Effect of Pruning on Adversarial Robustness Artur Jordão Hélio Pedrini AAML 111 23 0 10 Aug 2021
Differentiable Subset Pruning of Transformer Heads Jiaoda Li Ryan Cotterell Mrinmaya Sachan 134 57 0 10 Aug 2021
Exceeding the Limits of Visual-Linguistic Multi-Task Learning Cameron R. Wolfe Keld T. Lundgaard VLM 76 2 0 27 Jul 2021
Only Train Once: A One-Shot Neural Network Training And Pruning Framework Tianyi Chen Bo Ji Tianyu Ding Biyi Fang Guanyi Wang Zhihui Zhu Luming Liang Yixin Shi Sheng Yi Xiao Tu 127 105 0 15 Jul 2021
Learned Token Pruning for Transformers Sehoon Kim Sheng Shen D. Thorsley A. Gholami Woosuk Kwon Joseph Hassoun Kurt Keutzer 84 157 0 02 Jul 2021
Elbert: Fast Albert with Confidence-Window Based Early Exit Keli Xie Siyuan Lu Meiqi Wang Zhongfeng Wang 54 20 0 01 Jul 2021
Stateful ODE-Nets using Basis Function Expansions A. Queiruga N. Benjamin Erichson Liam Hodgkinson Michael W. Mahoney 63 16 0 21 Jun 2021
Layer Pruning on Demand with Intermediate CTC Jaesong Lee Jingu Kang Shinji Watanabe 40 18 0 17 Jun 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 174 863 0 14 Jun 2021
Chasing Sparsity in Vision Transformers: An End-to-End Exploration Tianlong Chen Yu Cheng Zhe Gan Lu Yuan Lei Zhang Zhangyang Wang ViT 70 224 0 08 Jun 2021
RoSearch: Search for Robust Student Architectures When Distilling Pre-trained Language Models Xin Guo Jianlei Yang Haoyi Zhou Xucheng Ye Jianxin Li 52 1 0 07 Jun 2021
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings Tatiana Likhomanenko Qiantong Xu Gabriel Synnaeve R. Collobert A. Rogozhnikov OOD ViT 84 60 0 06 Jun 2021
You Only Compress Once: Towards Effective and Elastic BERT Compression via Exploit-Explore Stochastic Nature Gradient Shaokun Zhang Xiawu Zheng Chenyi Yang Yuchao Li Yan Wang Chia-Wen Lin Mengdi Wang Shen Li Jun Yang Rongrong Ji MQ 97 23 0 04 Jun 2021
DoT: An efficient Double Transformer for NLP tasks with tables Syrine Krichene Thomas Müller Julian Martin Eisenschlos 73 14 0 01 Jun 2021
Greedy-layer Pruning: Speeding up Transformer Models for Natural Language Processing David Peer Sebastian Stabinger Stefan Engl A. Rodríguez-Sánchez 49 28 0 31 May 2021
LEAP: Learnable Pruning for Transformer-based Models Z. Yao Xiaoxia Wu Linjian Ma Sheng Shen Kurt Keutzer Michael W. Mahoney Yuxiong He 62 7 0 30 May 2021
NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search Jin Xu Xu Tan Renqian Luo Kaitao Song Jian Li Tao Qin Tie-Yan Liu MQ 62 79 0 30 May 2021
Accelerating BERT Inference for Sequence Labeling via Early-Exit Xiaonan Li Yunfan Shao Tianxiang Sun Hang Yan Xipeng Qiu Xuanjing Huang 84 41 0 28 May 2021
Early Exiting with Ensemble Internal Classifiers Tianxiang Sun Yunhua Zhou Xiangyang Liu Xinyu Zhang Hao Jiang Bo Zhao Xuanjing Huang Xipeng Qiu 70 31 0 28 May 2021
Inspecting the concept knowledge graph encoded by modern language models Carlos Aspillaga Marcelo Mendoza Alvaro Soto 72 13 0 27 May 2021
Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization Chen Liang Simiao Zuo Minshuo Chen Haoming Jiang Xiaodong Liu Pengcheng He T. Zhao Weizhu Chen 59 69 0 25 May 2021
TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference Deming Ye Yankai Lin Yufei Huang Maosong Sun MQ 84 65 0 25 May 2021
BERT Busters: Outlier Dimensions that Disrupt Transformers Olga Kovaleva Saurabh Kulshreshtha Anna Rogers Anna Rumshisky 117 92 0 14 May 2021
Not All Memories are Created Equal: Learning to Forget by Expiring Sainbayar Sukhbaatar Da Ju Spencer Poff Stephen Roller Arthur Szlam Jason Weston Angela Fan CLL 75 35 0 13 May 2021