Why Skip If You Can Combine: A Simple Knowledge Distillation Technique
for Intermediate Layers

Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers

6 October 2020

Mehdi Rezagholizade

Qun Liu

ArXiv (abs)PDF HTML

Papers citing "Why Skip If You Can Combine: A Simple Knowledge Distillation Technique for Intermediate Layers"

17 / 17 papers shown

Title
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 118 817 0 06 Apr 2020
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 113 1,872 0 23 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 142 843 0 25 Aug 2019
What Does BERT Look At? An Analysis of BERT's Attention Kevin Clark Urvashi Khandelwal Omer Levy Christopher D. Manning MILM 235 1,605 0 11 Jun 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 126 3,156 0 01 Apr 2019
Multilingual Neural Machine Translation with Knowledge Distillation Xu Tan Yi Ren Di He Tao Qin Zhou Zhao Tie-Yan Liu 93 250 0 27 Feb 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,229 0 11 Oct 2018
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 206 3,531 0 19 Aug 2018
A Call for Clarity in Reporting BLEU Scores Matt Post 181 2,998 0 23 Apr 2018
Marian: Fast Neural Machine Translation in C++ Marcin Junczys-Dowmunt Roman Grundkiewicz Tomasz Dwojak Hieu T. Hoang Kenneth Heafield ... Ulrich Germann Alham Fikri Aji Nikolay Bogoychev André F. T. Martins Alexandra Birch 98 718 0 01 Apr 2018
Tensor2Tensor for Neural Machine Translation Ashish Vaswani Samy Bengio E. Brevdo François Chollet Aidan Gomez ... Nal Kalchbrenner Niki Parmar Ryan Sepassi Noam M. Shazeer Jakob Uszkoreit 98 530 0 16 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 803 132,454 0 12 Jun 2017
Ensemble Distillation for Neural Machine Translation Markus Freitag Yaser Al-Onaizan B. Sankaran FedML 67 111 0 06 Feb 2017
OpenNMT: Open-Source Toolkit for Neural Machine Translation Guillaume Klein Yoon Kim Yuntian Deng Jean Senellart Alexander M. Rush 330 1,900 0 10 Jan 2017
Sequence-Level Knowledge Distillation Yoon Kim Alexander M. Rush 130 1,123 0 25 Jun 2016
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 367 19,745 0 09 Mar 2015
Neural Machine Translation by Jointly Learning to Align and Translate Dzmitry Bahdanau Kyunghyun Cho Yoshua Bengio AIMat 578 27,338 0 01 Sep 2014

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.