One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers

2 June 2021

Papers citing "One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teachers"

20 / 20 papers shown

Title
MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU Yan Li So-Eon Kim Seong-Bae Park S. Han 80 1 0 15 Aug 2024
EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation Yuqiao Wen Behzad Shayegh Chenyang Huang Yanshuai Cao Lili Mou 105 5 0 29 Feb 2024
Adaptive Multi-Teacher Multi-level Knowledge Distillation Yuang Liu Wei Zhang Jun Wang 70 157 0 06 Mar 2021
NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application Chuhan Wu Fangzhao Wu Yang Yu Tao Qi Yongfeng Huang Qi Liu VLM 46 45 0 09 Feb 2021
Reinforced Multi-Teacher Selection for Knowledge Distillation Fei Yuan Linjun Shou J. Pei Wutao Lin Ming Gong Yan Fu Daxin Jiang 56 122 0 11 Dec 2020
Improving Attention Mechanism with Query-Value Interaction Chuhan Wu Fangzhao Wu Tao Qi Yongfeng Huang 37 4 0 08 Oct 2020
Investigating Gender Bias in BERT Rishabh Bhardwaj Navonil Majumder Soujanya Poria 64 108 0 10 Sep 2020
Knowledge Distillation: A Survey Jianping Gou B. Yu Stephen J. Maybank Dacheng Tao VLM 110 2,976 0 09 Jun 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 366 1,489 0 18 Mar 2020
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training Hangbo Bao Li Dong Furu Wei Wenhui Wang Nan Yang ... Yu Wang Songhao Piao Jianfeng Gao Ming Zhou H. Hon AI4CE 88 394 0 28 Feb 2020
MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers Wenhui Wang Furu Wei Li Dong Hangbo Bao Nan Yang Ming Zhou VLM 156 1,278 0 25 Feb 2020
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 234 7,547 0 02 Oct 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 109 1,869 0 23 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 134 843 0 25 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 674 24,528 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 234 8,444 0 19 Jun 2019
Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao M. Zhou H. Hon ELM AI4CE 227 1,559 0 08 May 2019
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks Raphael Tang Yao Lu Linqing Liu Lili Mou Olga Vechtomova Jimmy J. Lin 69 421 0 28 Mar 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,114 0 11 Oct 2018
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 2.0K 150,260 0 22 Dec 2014