Extract then Distill: Efficient and Effective Task-Agnostic BERT
Distillation

Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation

24 April 2021

Lifeng Shang

Xin Jiang

Qun Liu

Papers citing "Extract then Distill: Efficient and Effective Task-Agnostic BERT Distillation"

7 / 7 papers shown

Title
VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning Oshin Dutta Ritvik Gupta Sumeet Agarwal 49 2 0 07 Jun 2024
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 32 24 0 19 Feb 2023
Gradient-based Intra-attention Pruning on Pre-trained Language Models Ziqing Yang Yiming Cui Xin Yao Shijin Wang VLM 42 8 0 15 Dec 2022
bert2BERT: Towards Reusable Pretrained Language Models Cheng Chen Yichun Yin Lifeng Shang Xin Jiang Yujia Qin Fengyu Wang Zhi Wang Xiao Chen Zhiyuan Liu Qun Liu VLM 29 59 0 14 Oct 2021
AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Models Yichun Yin Cheng Chen Lifeng Shang Xin Jiang Xiao Chen Qun Liu VLM 22 50 0 29 Jul 2021
Optimal Subarchitecture Extraction For BERT Adrian de Wynter Daniel J. Perry MQ 56 18 0 20 Oct 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 304 7,005 0 20 Apr 2018