Sequence-Level Knowledge Distillation

25 June 2016

Papers citing "Sequence-Level Knowledge Distillation"

50 / 244 papers shown

Title
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Zhigang Wang Shi Wang Qianqian Xu Qingming Huang 42 0 0 07 May 2025
Group Relative Knowledge Distillation: Learning from Teacher's Relational Inductive Bias Chao Li Changhua Zhou Jia Chen 24 0 0 29 Apr 2025
KETCHUP: K-Step Return Estimation for Sequential Knowledge Distillation Jiabin Fan Guoqing Luo Michael Bowling Lili Mou OffRL 68 0 0 26 Apr 2025
Teach Me How to Denoise: A Universal Framework for Denoising Multi-modal Recommender Systems via Guided Calibration Yiming Li Hanwen Du Yongbin Li Junchen Fu Chunxiao Li Ziyi Zhuang Jiakang Li Yongxin Ni AI4TS 29 0 0 19 Apr 2025
From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs Jiliang Ni Jiachen Pu Zhongyi Yang Kun Zhou Hui Wang Xiaoliang Xiao Dakui Wang Xin Li Jingfeng Luo Conggang Hu 39 0 0 18 Apr 2025
LLaVE: Large Language and Vision Embedding Models with Hardness-Weighted Contrastive Learning Zhibin Lan Liqiang Niu Fandong Meng Jie Zhou Jinsong Su VLM 69 2 0 04 Mar 2025
FourierNAT: A Fourier-Mixing-Based Non-Autoregressive Transformer for Parallel Sequence Generation Andrew Kiruluta Eric Lundy Andreas Lemos AI4TS 47 0 0 04 Mar 2025
Advantage-Guided Distillation for Preference Alignment in Small Language Models Shiping Gao Fanqi Wan Jiajian Guo Xiaojun Quan Qifan Wang ALM 58 0 0 25 Feb 2025
Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation Hieu Nguyen Zihao He Shoumik Atul Gandre Ujjwal Pasupulety Sharanya Kumari Shivakumar Kristina Lerman HILM 59 1 0 16 Feb 2025
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs Nicolas Boizard Kevin El Haddad C´eline Hudelot Pierre Colombo 77 15 0 28 Jan 2025
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models Makoto Shing Kou Misaki Han Bao Sho Yokoi Takuya Akiba VLM 57 1 0 28 Jan 2025
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion Zhaoyi Yan Zhijie Sang Yiming Zhang Yuhao Fu Baoyi He Qi Zhou Yining Di Chunlin Ji Shengyu Zhang Fei Wu MoMe LRM 64 1 0 06 Jan 2025
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison Tsz Kin Lam Marco Gaido Sara Papi L. Bentivogli Barry Haddow 36 0 0 04 Jan 2025
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models Xiao Cui Mo Zhu Yulei Qin Liang Xie Wengang Zhou Yiming Li 91 4 0 19 Dec 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
SWITCH: Studying with Teacher for Knowledge Distillation of Large Language Models Jahyun Koo Yerin Hwang Yongil Kim Taegwan Kang Hyunkyung Bae Kyomin Jung 60 0 0 25 Oct 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 73 5 0 22 Oct 2024
Speculative Knowledge Distillation: Bridging the Teacher-Student Gap Through Interleaved Sampling Wenyuan Xu Rujun Han Zhenting Wang L. Le Dhruv Madeka Lei Li Luu Anh Tuan Rishabh Agarwal Chen-Yu Lee Tomas Pfister 80 8 0 15 Oct 2024
Efficient Inference for Large Language Model-based Generative Recommendation Xinyu Lin Chaoqun Yang Wenjie Wang Yongqi Li Cunxiao Du Fuli Feng See-Kiong Ng Tat-Seng Chua 70 4 0 07 Oct 2024
HarmAug: Effective Data Augmentation for Knowledge Distillation of Safety Guard Models Seanie Lee Haebin Seong Dong Bok Lee Minki Kang Xiaoyin Chen Dominik Wagner Yoshua Bengio Juho Lee Sung Ju Hwang 67 2 0 02 Oct 2024
Explanation Bottleneck Models Shinýa Yamaguchi Kosuke Nishida LRM BDL 51 1 0 26 Sep 2024
Beyond Next Token Prediction: Patch-Level Training for Large Language Models Chenze Shao Fandong Meng Jie Zhou 51 1 0 17 Jul 2024
Don't Throw Away Data: Better Sequence Knowledge Distillation Jun Wang Eleftheria Briakou Hamid Dadkhahi Rishabh Agarwal Colin Cherry Trevor Cohn 53 5 0 15 Jul 2024
Direct Preference Knowledge Distillation for Large Language Models Yixing Li Yuxian Gu Li Dong Dequan Wang Yu Cheng Furu Wei 45 6 0 28 Jun 2024
Factual Dialogue Summarization via Learning from Large Language Models Rongxin Zhu Jey Han Lau Jianzhong Qi HILM 52 1 0 20 Jun 2024
BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM Zhewen Shen Aditya Joshi Ruey-Cheng Chen CLL 52 2 0 17 Jun 2024
Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models Dominik Wagner Ilja Baumann Korbinian Riedhammer Tobias Bocklet MQ 34 1 0 16 Jun 2024
CTC-based Non-autoregressive Textless Speech-to-Speech Translation Qingkai Fang Zhengrui Ma Yan Zhou Min Zhang Yang Feng 52 0 0 11 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation Zhengrui Ma Qingkai Fang Shaolei Zhang Shoutao Guo Yang Feng Min Zhang 53 9 0 11 Jun 2024
Label-Synchronous Neural Transducer for E2E Simultaneous Speech Translation Keqi Deng Philip C. Woodland 43 4 0 06 Jun 2024
$$\textit{Trans-LoRA}$: towards data-free Transferable Parameter Efficient Finetuning$ $\textit{Trans-LoRA}$ : towards data-free Transferable Parameter Efficient Finetuning Runqian Wang Soumya Ghosh David D. Cox Diego Antognini Aude Oliva Rogerio Feris Leonid Karlinsky 40 1 0 27 May 2024
Sentence-Level or Token-Level? A Comprehensive Study on Knowledge Distillation Jingxuan Wei Linzhuang Sun Yichong Leng Xu Tan Bihui Yu Ruifeng Guo 51 3 0 23 Apr 2024
Parameter Efficient Diverse Paraphrase Generation Using Sequence-Level Knowledge Distillation Lasal Jayawardena Prasan Yapa BDL 38 1 0 19 Apr 2024
Towards Multimodal Video Paragraph Captioning Models Robust to Missing Modality Sishuo Chen Lei Li Shuhuai Ren Rundong Gao Yuanxin Liu Xiaohan Bi Xu Sun Lu Hou 57 3 0 28 Mar 2024
The Unreasonable Ineffectiveness of the Deeper Layers Andrey Gromov Kushal Tirumala Hassan Shapourian Paolo Glorioso Daniel A. Roberts 52 81 0 26 Mar 2024
EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation Yuqiao Wen Behzad Shayegh Chenyang Huang Yanshuai Cao Lili Mou 50 4 0 29 Feb 2024
Analysis of Levenshtein Transformer's Decoder and Its Variants Ruiyang Zhou 14 0 0 19 Feb 2024
Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages Yuan Zhang Yile Wang Zijun Liu Shuo Wang Xiaolong Wang Peng Li Maosong Sun Yang Liu LRM 35 11 0 19 Feb 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 52 252 0 19 Jan 2024
Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decoding Heming Xia Zhe Yang Qingxiu Dong Peiyi Wang Yongqi Li Tao Ge Tianyu Liu Wenjie Li Zhifang Sui LRM 38 101 0 15 Jan 2024
Multi-teacher Distillation for Multilingual Spelling Correction Jingfen Zhang Xuan Guo S. Bodapati Christopher Potts KELM 27 3 0 20 Nov 2023
DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency Azhar Shaikh Michael Cochez Denis Diachkov Michiel de Rijcke Sahar Yousefi 27 0 0 09 Nov 2023
Non-autoregressive Streaming Transformer for Simultaneous Translation Zhengrui Ma Shaolei Zhang Shoutao Guo Chenze Shao Min Zhang Yang Feng 32 13 0 23 Oct 2023
Tuna: Instruction Tuning using Feedback from Large Language Models Haoran Li Yiran Liu Xingxing Zhang Wei Lu Furu Wei ALM 38 3 0 20 Oct 2023
A Survey of Imbalanced Learning on Graphs: Problems, Techniques, and Future Directions Zemin Liu Yuan N. Li Nan-Fang Chen Qian Wang Bryan Hooi Bin He FaML 14 13 0 26 Aug 2023
Teacher-Student Architecture for Knowledge Distillation: A Survey Chengming Hu Xuan Li Danyang Liu Haolun Wu Xi Chen Ju Wang Xue Liu 21 16 0 08 Aug 2023
Learning Evaluation Models from Large Language Models for Sequence Generation Chenglong Wang Hang Zhou Kai-Chun Chang Tongran Liu Chunliang Zhang Quan Du Tong Xiao Yue Zhang Jingbo Zhu ELM 46 3 0 08 Aug 2023
f-Divergence Minimization for Sequence-Level Knowledge Distillation Yuqiao Wen Zichao Li Wenyu Du Lili Mou 32 53 0 27 Jul 2023
RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment Kevin Kaichuang Yang Dan Klein Asli Celikyilmaz Nanyun Peng Yuandong Tian ALM 36 30 0 24 Jul 2023
Intuitive Access to Smartphone Settings Using Relevance Model Trained by Contrastive Learning Joonyoung Kim Kangwook Lee Haebin Shin Hurnjoo Lee Sechun Kang Byunguk Choi Dong Shin Joohyung Lee 25 0 0 15 Jul 2023