Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method

11 June 2023

Papers citing "Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method"

7 / 7 papers shown

Title
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 253 1,073 0 05 Oct 2022
Causal Distillation for Language Models Zhengxuan Wu Atticus Geiger J. Rozner Elisa Kreiss Hanson Lu Thomas Icard Christopher Potts Noah D. Goodman 89 25 0 05 Dec 2021
HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression Chenhe Dong Yaliang Li Ying Shen Minghui Qiu VLM 34 7 0 16 Oct 2021
Distilling Linguistic Context for Language Model Compression Geondo Park Gyeongman Kim Eunho Yang 45 38 0 17 Sep 2021
LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding Hao Fu Shaojun Zhou Qihong Yang Junjie Tang Guiquan Liu Kaikui Liu Xiaolong Li 37 57 0 14 Dec 2020
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing Canwen Xu Wangchunshu Zhou Tao Ge Furu Wei Ming Zhou 221 197 0 07 Feb 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018