TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference

25 May 2021

Deming Ye

Yankai Lin

Yufei Huang

Maosong Sun

ArXiv PDF HTML

Papers citing "TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference"

41 / 41 papers shown

Title
Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search Gyuwan Kim Kyunghyun Cho 69 96 0 14 Oct 2020
TernaryBERT: Distillation-aware Ultra-low Bit BERT Wei Zhang Lu Hou Yichun Yin Lifeng Shang Xiao Chen Xin Jiang Qun Liu MQ 76 210 0 27 Sep 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 499 2,074 0 28 Jul 2020
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing Zihang Dai Guokun Lai Yiming Yang Quoc V. Le 76 233 0 05 Jun 2020
DeFormer: Decomposing Pre-trained Transformers for Faster Question Answering Qingqing Cao H. Trivedi A. Balasubramanian Niranjan Balasubramanian 63 68 0 02 May 2020
DeeBERT: Dynamic Early Exiting for Accelerating BERT Inference Ji Xin Raphael Tang Jaejun Lee Yaoliang Yu Jimmy J. Lin 53 372 0 27 Apr 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 124 4,048 0 10 Apr 2020
DynaBERT: Dynamic BERT with Adaptive Width and Depth Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen Qun Liu MQ 73 322 0 08 Apr 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 97 811 0 06 Apr 2020
FastBERT: a Self-distilling BERT with Adaptive Inference Time Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Haotang Deng Qi Ju 82 359 0 05 Apr 2020
A Primer in BERTology: What we know about how BERT works Anna Rogers Olga Kovaleva Anna Rumshisky OffRL 80 1,494 0 27 Feb 2020
PoWER-BERT: Accelerating BERT Inference via Progressive Word-vector Elimination Saurabh Goyal Anamitra R. Choudhury Saurabh ManishRaje Venkatesan T. Chakaravarthy Yogish Sabharwal Ashish Verma 54 18 0 24 Jan 2020
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension Adam Fisch Alon Talmor Robin Jia Minjoon Seo Eunsol Choi Danqi Chen 59 306 0 22 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 183 7,465 0 02 Oct 2019
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 105 591 0 25 Sep 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 85 1,855 0 23 Sep 2019
Revealing the Importance of Semantic Retrieval for Machine Reading at Scale Yixin Nie Songhe Wang Joey Tianyi Zhou RALM 200 136 0 17 Sep 2019
Semantics-aware BERT for Language Understanding Zhuosheng Zhang Yuwei Wu Zhao Hai Z. Li Shuailiang Zhang Xi Zhou Xiang Zhou 40 368 0 05 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 114 836 0 25 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 498 24,351 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 211 8,415 0 19 Jun 2019
Open Sesame: Getting Inside BERT's Linguistic Knowledge Yongjie Lin Y. Tan Robert Frank 50 287 0 04 Jun 2019
MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehension Alon Talmor Jonathan Berant 67 173 0 31 May 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 95 1,058 0 25 May 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 90 1,894 0 23 Apr 2019
Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence Chi Sun Luyao Huang Xipeng Qiu 61 610 0 22 Mar 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.3K 94,511 0 11 Oct 2018
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W. Cohen Ruslan Salakhutdinov Christopher D. Manning RALM 143 2,635 0 25 Sep 2018
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 76 752 0 10 Jul 2018
Know What You Don't Know: Unanswerable Questions for SQuAD Pranav Rajpurkar Robin Jia Percy Liang RALM ELM 237 2,835 0 11 Jun 2018
Simple and Effective Multi-Paragraph Reading Comprehension Christopher Clark Matt Gardner RALM 70 458 0 29 Oct 2017
Constructing Datasets for Multi-hop Reading Comprehension Across Documents Johannes Welbl Pontus Stenetorp Sebastian Riedel SyDa RALM 87 509 0 17 Oct 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 591 130,942 0 12 Jun 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 195 2,636 0 09 May 2017
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 158 1,343 0 15 Apr 2017
NewsQA: A Machine Comprehension Dataset Adam Trischler Tong Wang Xingdi Yuan Justin Harris Alessandro Sordoni Philip Bachman Kaheer Suleman 82 893 0 29 Nov 2016
A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks Dan Hendrycks Kevin Gimpel UQCV 128 3,441 0 07 Oct 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 215 8,113 0 16 Jun 2016
Character-level Convolutional Networks for Text Classification Xiang Zhang Jiaqi Zhao Yann LeCun 234 6,100 0 04 Sep 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 300 19,580 0 09 Mar 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.3K 149,842 0 22 Dec 2014