KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation

13 September 2021

Papers citing "KroneckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowledge Distillation"

27 / 27 papers shown

Title
FastFormers: Highly Efficient Transformer Models for Natural Language Understanding Young Jin Kim Hany Awadalla AI4CE 63 44 0 26 Oct 2020
Contrastive Distillation on Intermediate Representations for Language Model Compression S. Sun Zhe Gan Yu Cheng Yuwei Fang Shuohang Wang Jingjing Liu VLM 49 70 0 29 Sep 2020
Pretrained Transformers Improve Out-of-Distribution Robustness Dan Hendrycks Xiaoyuan Liu Eric Wallace Adam Dziedzic R. Krishnan D. Song OOD 151 434 0 13 Apr 2020
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression Yihuan Mao Yujing Wang Chufan Wu Chen Zhang Yang-Feng Wang Yaming Yang Quanlu Zhang Yunhai Tong Jing Bai 44 73 0 08 Apr 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 95 811 0 06 Apr 2020
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 255 201 0 07 Feb 2020
Compressing Language Models using Doped Kronecker Products Urmish Thakker Paul Whatamough Zhi-Gang Liu Matthew Mattina Jesse G. Beu 26 9 0 24 Jan 2020
Fully Quantized Transformer for Machine Translation Gabriele Prato Ella Charlaix Mehdi Rezagholizadeh MQ 40 70 0 17 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 178 7,465 0 02 Oct 2019
Extremely Small BERT Models from Mixed-Vocabulary Training Sanqiang Zhao Raghav Gupta Yang Song Denny Zhou VLM 42 53 0 25 Sep 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 83 1,855 0 23 Sep 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 308 1,892 0 17 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 112 836 0 25 Aug 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 209 8,415 0 19 Jun 2019
PAWS: Paraphrase Adversaries from Word Scrambling Yuan Zhang Jason Baldridge Luheng He 68 543 0 01 Apr 2019
Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference R. Thomas McCoy Ellie Pavlick Tal Linzen 127 1,234 0 04 Feb 2019
Tensorized Embedding Layers for Efficient Model Compression Oleksii Hrinchuk Valentin Khrulkov L. Mirvakhabova Elena Orlova Ivan Oseledets 54 73 0 30 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.3K 94,511 0 11 Oct 2018
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 173 3,514 0 19 Aug 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 820 7,141 0 20 Apr 2018
Slim Embedding Layers for Recurrent Neural Language Models Zhongliang Li Raymond Kulhanek Shaojun Wang Yunxin Zhao Shuang Wu KELM 52 23 0 27 Nov 2017
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 207 8,113 0 16 Jun 2016
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 227 8,821 0 01 Oct 2015
Compression of Fully-Connected Layer in Neural Network by Kronecker Product Shuchang Zhou Jia-Nan Wu 52 26 0 21 Jul 2015
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 116 2,544 0 22 Jun 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 300 19,580 0 09 Mar 2015
Compressing Deep Convolutional Networks using Vector Quantization Yunchao Gong Liu Liu Ming Yang Lubomir D. Bourdev MQ 120 1,170 0 18 Dec 2014