Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 400 papers shown

Title
Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Prior Zi Lin Jeremiah Zhe Liu Ziao Yang Nan Hua Dan Roth 25 46 0 05 Oct 2020
Which *BERT? A Survey Organizing Contextualized Encoders Patrick Xia Shijie Wu Benjamin Van Durme 26 50 0 02 Oct 2020
AUBER: Automated BERT Regularization Hyun Dong Lee Seongmin Lee U. Kang 16 7 0 30 Sep 2020
Deep Transformers with Latent Depth Xian Li Asa Cooper Stickland Yuqing Tang X. Kong 19 22 0 28 Sep 2020
TernaryBERT: Distillation-aware Ultra-low Bit BERT Wei Zhang Lu Hou Yichun Yin Lifeng Shang Xiao Chen Xin Jiang Qun Liu MQ 33 208 0 27 Sep 2020
Alleviating the Inequality of Attention Heads for Neural Machine Translation Zewei Sun Shujian Huang Xinyu Dai Jiajun Chen 13 7 0 21 Sep 2020
Dissecting Lottery Ticket Transformers: Structural and Behavioral Study of Sparse Neural Machine Translation Rajiv Movva Jason Zhao 18 12 0 17 Sep 2020
Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation M. Tukan Alaa Maalouf Matan Weksler Dan Feldman 15 9 0 11 Sep 2020
Compression of Deep Learning Models for Text: A Survey Manish Gupta Puneet Agrawal VLM MedIm AI4CE 17 115 0 12 Aug 2020
ConvBERT: Improving BERT with Span-based Dynamic Convolution Zihang Jiang Weihao Yu Daquan Zhou Yunpeng Chen Jiashi Feng Shuicheng Yan 37 157 0 06 Aug 2020
Compressing Deep Neural Networks via Layer Fusion James OÑeill Greg Ver Steeg Aram Galstyan AI4CE 14 4 0 29 Jul 2020
Contrastive Visual-Linguistic Pretraining Lei Shi Kai Shuang Shijie Geng Peng Su Zhengkai Jiang Peng Gao Zuohui Fu Gerard de Melo Sen Su VLM SSL CLIP 33 29 0 26 Jul 2020
Diverse Ensembles Improve Calibration Asa Cooper Stickland Iain Murray UQCV FedML 27 26 0 08 Jul 2020
PyTorch Distributed: Experiences on Accelerating Data Parallel Training Shen Li Yanli Zhao R. Varma Omkar Salpekar P. Noordhuis ... Adam Paszke Jeff Smith Brian Vaughan Pritam Damania Soumith Chintala OOD MoE 8 178 0 28 Jun 2020
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions Stephen Roller Y-Lan Boureau Jason Weston Antoine Bordes Emily Dinan ... Kurt Shuster Eric Michael Smith Arthur Szlam Jack Urbanek Mary Williamson LLMAG AI4CE 28 51 0 22 Jun 2020
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations Alexei Baevski Henry Zhou Abdel-rahman Mohamed Michael Auli SSL 8 5,553 0 20 Jun 2020
Multi-branch Attentive Transformer Yang Fan Shufang Xie Yingce Xia Lijun Wu Tao Qin Xiang-Yang Li Tie-Yan Liu 11 17 0 18 Jun 2020
BERT Loses Patience: Fast and Robust Inference with Early Exit Wangchunshu Zhou Canwen Xu Tao Ge Julian McAuley Ke Xu Furu Wei 6 329 0 07 Jun 2020
Normalized Attention Without Probability Cage Oliver Richter Roger Wattenhofer 14 21 0 19 May 2020
Iterative Pseudo-Labeling for Speech Recognition Qiantong Xu Tatiana Likhomanenko Jacob Kahn Awni Y. Hannun Gabriel Synnaeve R. Collobert VLM 21 129 0 19 May 2020
Movement Pruning: Adaptive Sparsity by Fine-Tuning Victor Sanh Thomas Wolf Alexander M. Rush 32 466 0 15 May 2020
Adaptive Transformers for Learning Multimodal Representations Prajjwal Bhargava 14 4 0 15 May 2020
A Mixture of $h-1$ Heads is Better than $h$ Heads Hao Peng Roy Schwartz Dianqi Li Noah A. Smith MoE 27 32 0 13 May 2020
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference Ali Hadi Zadeh Isak Edo Omar Mohamed Awad Andreas Moshovos MQ 30 183 0 08 May 2020
MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases Louis Martin Angela Fan Eric Villemonte de la Clergerie Antoine Bordes Benoît Sagot 20 36 0 01 May 2020
Scheduled DropHead: A Regularization Method for Transformer Models Wangchunshu Zhou Tao Ge Ke Xu Furu Wei Ming Zhou 12 36 0 28 Apr 2020
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference Ji Xin Raphael Tang Jaejun Lee Yaoliang Yu Jimmy J. Lin 6 363 0 27 Apr 2020
Faster Depth-Adaptive Transformers Yijin Liu Fandong Meng Jie Zhou Yufeng Chen Jinan Xu 14 2 0 27 Apr 2020
The Right Tool for the Job: Matching Model and Instance Complexities Roy Schwartz Gabriel Stanovsky Swabha Swayamdipta Jesse Dodge Noah A. Smith 33 167 0 16 Apr 2020
Training with Quantization Noise for Extreme Model Compression Angela Fan Pierre Stock Benjamin Graham Edouard Grave Remi Gribonval Hervé Jégou Armand Joulin MQ 22 242 0 15 Apr 2020
On Optimal Transformer Depth for Low-Resource Language Translation Elan Van Biljon Arnu Pretorius Julia Kreutzer MoE 19 27 0 09 Apr 2020
DynaBERT: Dynamic BERT with Adaptive Width and Depth Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen Qun Liu MQ 15 319 0 08 Apr 2020
On the Effect of Dropping Layers of Pre-trained Transformer Models Hassan Sajjad Fahim Dalvi Nadir Durrani Preslav Nakov 31 132 0 08 Apr 2020
PowerNorm: Rethinking Batch Normalization in Transformers Sheng Shen Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer BDL 18 16 0 17 Mar 2020
A Primer in BERTology: What we know about how BERT works Anna Rogers Olga Kovaleva Anna Rumshisky OffRL 35 1,456 0 27 Feb 2020
Compressing Large-Scale Transformer-Based Models: A Case Study on BERT Prakhar Ganesh Yao Chen Xin Lou Mohammad Ali Khan Xuming Hu Hassan Sajjad Preslav Nakov Deming Chen Marianne Winslett AI4CE 16 197 0 27 Feb 2020
Addressing Some Limitations of Transformers with Feedback Memory Angela Fan Thibaut Lavril Edouard Grave Armand Joulin Sainbayar Sukhbaatar 26 11 0 21 Feb 2020
Controlling Computation versus Quality for Neural Sequence Models Ankur Bapna N. Arivazhagan Orhan Firat 19 30 0 17 Feb 2020
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 221 197 0 07 Feb 2020
Scaling Up Online Speech Recognition Using ConvNets Vineel Pratap Qiantong Xu Jacob Kahn Gilad Avidov Tatiana Likhomanenko Awni Y. Hannun Vitaliy Liptchinsky Gabriel Synnaeve R. Collobert 154 38 0 27 Jan 2020
BERT's output layer recognizes all hidden layers? Some Intriguing Phenomena and a simple way to boost BERT Wei-Tsung Kao Tsung-Han Wu Po-Han Chi Chun-Cheng Hsieh Hung-yi Lee SSL 18 5 0 25 Jan 2020
FlauBERT: Unsupervised Language Model Pre-training for French Hang Le Loïc Vial Jibril Frej Vincent Segonne Maximin Coavoux Benjamin Lecouteux A. Allauzen Benoît Crabbé Laurent Besacier D. Schwab AI4CE 40 395 0 11 Dec 2019
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures Gabriel Synnaeve Qiantong Xu Jacob Kahn Tatiana Likhomanenko Edouard Grave Vineel Pratap Anuroop Sriram Vitaliy Liptchinsky R. Collobert SSL AI4TS 30 246 0 19 Nov 2019
CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEB Holger Schwenk Guillaume Wenzek Sergey Edunov Edouard Grave Armand Joulin 25 255 0 10 Nov 2019
Structured Pruning of a BERT-based Question Answering Model J. Scott McCarley Rishav Chakravarti Avirup Sil 15 53 0 14 Oct 2019
Structured Pruning of Large Language Models Ziheng Wang Jeremy Wohlwend Tao Lei 24 281 0 10 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 11 1,815 0 23 Sep 2019
DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning Xiawu Zheng Chenyi Yang Shaokun Zhang Yan Wang Baochang Zhang Yongjian Wu Yunsheng Wu Ling Shao Rongrong Ji 40 21 0 28 May 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018
Improving neural networks by preventing co-adaptation of feature detectors Geoffrey E. Hinton Nitish Srivastava A. Krizhevsky Ilya Sutskever Ruslan Salakhutdinov VLM 266 7,636 0 03 Jul 2012