Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 400 papers shown

Title
Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study Salah Zaiem Robin Algayres Titouan Parcollet S. Essid Mirco Ravanelli 52 14 0 12 Mar 2023
X-Pruner: eXplainable Pruning for Vision Transformers Lu Yu Wei Xiang ViT 11 48 0 08 Mar 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 37 20 0 07 Mar 2023
BPT: Binary Point Cloud Transformer for Place Recognition Zhixing Hou Yuzhang Shang Tian Gao Yan Yan MQ ViT 37 3 0 02 Mar 2023
Full Stack Optimization of Transformer Inference: a Survey Sehoon Kim Coleman Hooper Thanakul Wattanawong Minwoo Kang Ruohan Yan ... Qijing Huang Kurt Keutzer Michael W. Mahoney Y. Shao A. Gholami MQ 36 101 0 27 Feb 2023
Towards multi-task learning of speech and speaker recognition Nik Vaessen David A. van Leeuwen CVBM 22 0 0 24 Feb 2023
Speculative Decoding with Big Little Decoder Sehoon Kim K. Mangalam Suhong Moon Jitendra Malik Michael W. Mahoney A. Gholami Kurt Keutzer MoE 26 99 0 15 Feb 2023
Stitchable Neural Networks Zizheng Pan Jianfei Cai Bohan Zhuang 53 22 0 13 Feb 2023
Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models Mohammadreza Banaei Klaudia Bałazy Artur Kasymov R. Lebret Jacek Tabor Karl Aberer OffRL 21 0 0 08 Feb 2023
ZipLM: Inference-Aware Structured Pruning of Language Models Eldar Kurtic Elias Frantar Dan Alistarh MQ 25 24 0 07 Feb 2023
Towards energy-efficient Deep Learning: An overview of energy-efficient approaches along the Deep Learning Lifecycle Vanessa Mehlin Sigurd Schacht Carsten Lanquillon HAI MedIm 33 19 0 05 Feb 2023
UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers Dachuan Shi Chaofan Tao Ying Jin Zhendong Yang Chun Yuan Jiaqi Wang VLM ViT 28 38 0 31 Jan 2023
Exploring Attention Map Reuse for Efficient Transformer Neural Networks Kyuhong Shim Jungwook Choi Wonyong Sung ViT 24 3 0 29 Jan 2023
Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases Xiaoxia Wu Cheng-rong Li Reza Yazdani Aminabadi Z. Yao Yuxiong He MQ 19 19 0 27 Jan 2023
When Layers Play the Lottery, all Tickets Win at Initialization Artur Jordão George Correa de Araujo H. Maia Hélio Pedrini 13 3 0 25 Jan 2023
Adapting a Language Model While Preserving its General Knowledge Zixuan Ke Yijia Shao Haowei Lin Hu Xu Lei Shu Bin Liu KELM CLL VLM 36 20 0 21 Jan 2023
FlexiViT: One Model for All Patch Sizes Lucas Beyer Pavel Izmailov Alexander Kolesnikov Mathilde Caron Simon Kornblith Xiaohua Zhai Matthias Minderer Michael Tschannen Ibrahim M. Alabdulmohsin Filip Pavetić VLM 45 90 0 15 Dec 2022
Gradient-based Intra-attention Pruning on Pre-trained Language Models Ziqing Yang Yiming Cui Xin Yao Shijin Wang VLM 32 8 0 15 Dec 2022
Co-training $2^L$ Submodels for Visual Recognition Hugo Touvron Matthieu Cord Maxime Oquab Piotr Bojanowski Jakob Verbeek Hervé Jégou VLM 35 9 0 09 Dec 2022
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention Zineng Tang Jaemin Cho Jie Lei Joey Tianyi Zhou VLM 24 9 0 21 Nov 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 40 6 0 17 Nov 2022
MelHuBERT: A simplified HuBERT on Mel spectrograms Tzu-Quan Lin Hung-yi Lee Hao Tang SSL 32 13 0 17 Nov 2022
Fast and Accurate FSA System Using ELBERT: An Efficient and Lightweight BERT Siyuan Lu Chenchen Zhou Keli Xie Jun Lin Zhongfeng Wang 16 1 0 16 Nov 2022
A Survey for Efficient Open Domain Question Answering Qin Zhang Shan Chen Dongkuan Xu Qingqing Cao Xiaojun Chen Trevor Cohn Meng Fang 28 33 0 15 Nov 2022
FPT: Improving Prompt Tuning Efficiency via Progressive Training Yufei Huang Yujia Qin Huadong Wang Yichun Yin Maosong Sun Zhiyuan Liu Qun Liu VLM LRM 35 6 0 13 Nov 2022
Speech-to-Speech Translation For A Real-world Unwritten Language Peng-Jen Chen Ke M. Tran Yilin Yang Jingfei Du Justine T. Kao ... Sravya Popuri Changhan Wang J. Pino Wei-Ning Hsu Ann Lee 34 26 0 11 Nov 2022
Bridging Fairness and Environmental Sustainability in Natural Language Processing Marius Hessenthaler Emma Strubell Dirk Hovy Anne Lauscher 24 8 0 08 Nov 2022
Streaming, fast and accurate on-device Inverse Text Normalization for Automatic Speech Recognition Yashesh Gaur Nick Kibre Jian Xue Kangyuan Shu Yuhui Wang Issac Alphonso Jinyu Li Jiawei Liu 16 6 0 07 Nov 2022
More Speaking or More Speakers? Dan Berrebbi R. Collobert Navdeep Jaitly Tatiana Likhomanenko 13 5 0 02 Nov 2022
Empirical Evaluation of Post-Training Quantization Methods for Language Tasks Ting Hu Christoph Meinel Haojin Yang MQ 28 3 0 29 Oct 2022
Efficient Speech Translation with Dynamic Latent Perceivers Ioannis Tsiamas Gerard I. Gállego José A. R. Fonollosa Marta R. Costa-jussá 17 2 0 28 Oct 2022
COST-EFF: Collaborative Optimization of Spatial and Temporal Efficiency with Slenderized Multi-exit Language Models Bowen Shen Zheng Lin Yuanxin Liu Zhengxiao Liu Lei Wang Weiping Wang VLM 47 4 0 27 Oct 2022
Real-time Speech Interruption Analysis: From Cloud to Client Deployment Quchen Fu Szu-Wei Fu Yaran Fan Yu-Huan Wu Zhuo Chen J. Gupchup Ross Cutler 34 0 0 24 Oct 2022
PATS: Sensitivity-aware Noisy Learning for Pretrained Language Models Yupeng Zhang Hongzhi Zhang Sirui Wang Wei Yu Wu Zhoujun Li AAML 33 1 0 22 Oct 2022
Named Entity Detection and Injection for Direct Speech Translation Marco Gaido Yun Tang Ilia Kulikov Rongqing Huang Hongyu Gong Hirofumi Inaguma 27 2 0 21 Oct 2022
Continuous Pseudo-Labeling from the Start Dan Berrebbi R. Collobert Samy Bengio Navdeep Jaitly Tatiana Likhomanenko 29 14 0 17 Oct 2022
Compute-Efficient Deep Learning: Algorithmic Trends and Opportunities Brian Bartoldson B. Kailkhura Davis W. Blalock 31 47 0 13 Oct 2022
Revisiting Structured Dropout Yiren Zhao Oluwatomisin Dada Xitong Gao Robert D. Mullins BDL 24 2 0 05 Oct 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 253 1,073 0 05 Oct 2022
Relaxed Attention for Transformer Models Timo Lohrenz Björn Möller Zhengyang Li Tim Fingscheidt KELM 29 11 0 20 Sep 2022
Parameter-Efficient Conformers via Sharing Sparsely-Gated Experts for End-to-End Speech Recognition Ye Bai Jie Li W. Han Hao Ni Kaituo Xu Zhuo Zhang Cheng Yi Xiaorui Wang MoE 23 1 0 17 Sep 2022
Efficient Methods for Natural Language Processing: A Survey Marcos Vinícius Treviso Ji-Ung Lee Tianchu Ji Betty van Aken Qingqing Cao ... Emma Strubell Niranjan Balasubramanian Leon Derczynski Iryna Gurevych Roy Schwartz 30 109 0 31 Aug 2022
FocusFormer: Focusing on What We Need via Architecture Sampler Jing Liu Jianfei Cai Bohan Zhuang 35 7 0 23 Aug 2022
Efficient model compression with Random Operation Access Specific Tile (ROAST) hashing Aditya Desai K. Zhou Anshumali Shrivastava 14 1 0 21 Jul 2022
Confident Adaptive Language Modeling Tal Schuster Adam Fisch Jai Gupta Mostafa Dehghani Dara Bahri Vinh Q. Tran Yi Tay Donald Metzler 43 160 0 14 Jul 2022
STI: Turbocharge NLP Inference at the Edge via Elastic Pipelining Liwei Guo Wonkyo Choe F. Lin 19 14 0 11 Jul 2022
Adversarial Self-Attention for Language Understanding Hongqiu Wu Ruixue Ding Hai Zhao Pengjun Xie Fei Huang Min Zhang 19 11 0 25 Jun 2022
PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importance Qingru Zhang Simiao Zuo Chen Liang Alexander Bukharin Pengcheng He Weizhu Chen T. Zhao 22 77 0 25 Jun 2022
Binary Early-Exit Network for Adaptive Inference on Low-Resource Devices Aaqib Saeed MQ 14 1 0 17 Jun 2022
LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning Yi-Lin Sung Jaemin Cho Joey Tianyi Zhou VLM 21 236 0 13 Jun 2022