Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation

24 February 2020

Yige Xu

Xipeng Qiu

L. Zhou

Xuanjing Huang

ArXiv PDF HTML

Papers citing "Improving BERT Fine-Tuning via Self-Ensemble and Self-Distillation"

31 / 31 papers shown

Title
Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data Jonathan Pilault Amine Elhattami C. Pal CLL MoE 38 89 0 19 Sep 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 73 807 0 06 Apr 2020
FastBERT: a Self-distilling BERT with Adaptive Inference Time Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Haotang Deng Qi Ju 66 356 0 05 Apr 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 306 1,471 0 18 Mar 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers Wenhui Wang Furu Wei Li Dong Hangbo Bao Nan Yang Ming Zhou VLM 61 1,230 0 25 Feb 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 175 19,824 0 23 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 54 7,386 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 30 1,838 0 23 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 88 833 0 25 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 309 24,160 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 148 8,386 0 19 Jun 2019
How to Fine-Tune BERT for Text Classification? Chi Sun Xipeng Qiu Yige Xu Xuanjing Huang 44 1,508 0 14 May 2019
Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao M. Zhou H. Hon ELM AI4CE 92 1,553 0 08 May 2019
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding Xiaodong Liu Pengcheng He Weizhu Chen Jianfeng Gao FedML 23 182 0 20 Apr 2019
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks Matthew E. Peters Sebastian Ruder Noah A. Smith 48 435 0 14 Mar 2019
Parameter-Efficient Transfer Learning for NLP N. Houlsby A. Giurgiu Stanislaw Jastrzebski Bruna Morrone Quentin de Laroussilhe Andrea Gesmundo Mona Attariyan Sylvain Gelly 149 4,368 0 02 Feb 2019
Multi-Task Deep Neural Networks for Natural Language Understanding Xiaodong Liu Pengcheng He Weizhu Chen Jianfeng Gao AI4CE 77 1,269 0 31 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 577 93,936 0 11 Oct 2018
Neural Network Acceptability Judgments Alex Warstadt Amanpreet Singh Samuel R. Bowman 112 1,390 0 31 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 421 7,080 0 20 Apr 2018
Distributed Prioritized Experience Replay Dan Horgan John Quan David Budden Gabriel Barth-Maron Matteo Hessel H. V. Hasselt David Silver 124 736 0 02 Mar 2018
SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation Daniel Cer Mona T. Diab Eneko Agirre I. Lopez-Gazpio Lucia Specia 67 1,870 0 31 Jul 2017
Hindsight Experience Replay Marcin Andrychowicz Dwight Crow Alex Ray Jonas Schneider Rachel Fong Peter Welinder Bob McGrew Joshua Tobin Pieter Abbeel Wojciech Zaremba OffRL 213 2,307 0 05 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 223 129,831 0 12 Jun 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 298 4,444 0 18 Apr 2017
Temporal Ensembling for Semi-Supervised Learning S. Laine Timo Aila UQCV 142 2,543 0 07 Oct 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 77 8,067 0 16 Jun 2016
Prioritized Experience Replay Tom Schaul John Quan Ioannis Antonoglou David Silver OffRL 171 3,777 0 18 Nov 2015
Character-level Convolutional Networks for Text Classification Xiang Zhang Jiaqi Zhao Yann LeCun 153 6,046 0 04 Sep 2015
A large annotated corpus for learning natural language inference Samuel R. Bowman Gabor Angeli Christopher Potts Christopher D. Manning 161 4,256 0 21 Aug 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 69 19,448 0 09 Mar 2015