Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs

28 January 2025

Papers citing "Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs"

20 / 20 papers shown

Title
ExpertSteer: Intervening in LLMs through Expert Knowledge Weixuan Wang Minghao Wu Barry Haddow Alexandra Birch LLMSV 7 0 0 18 May 2025
Feature Alignment and Representation Transfer in Knowledge Distillation for Large Language Models Junjie Yang Junhao Song Xudong Han Ziqian Bi Tianyang Wang ... Yujie Zhang Qian Niu Benji Peng Keyu Chen Ming Liu VLM 47 0 0 18 Apr 2025
A Dual-Space Framework for General Knowledge Distillation of Large Language Models Jiahui Geng Songming Zhang Yunlong Liang Fandong Meng Yufeng Chen Jinan Xu Jie Zhou 26 0 0 15 Apr 2025
Cross-Tokenizer Distillation via Approximate Likelihood Matching Benjamin Minixhofer Ivan Vulić Edoardo Ponti 193 0 0 25 Mar 2025
Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling Haebin Shin Lei Ji Xiao Liu Yeyun Gong 57 0 0 24 Mar 2025
Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs Anshumann Mohd Abbas Zaidi Akhil Kedia Jinwoo Ahn Taehwak Kwon Kangwook Lee Haejun Lee Joohyung Lee FedML 194 1 0 21 Mar 2025
Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models Gyeongman Kim Gyouk Chu Eunho Yang MoE 59 0 0 18 Feb 2025
Smoothing Out Hallucinations: Mitigating LLM Hallucination with Smoothed Knowledge Distillation Hieu Nguyen Zihao He Shoumik Atul Gandre Ujjwal Pasupulety Sharanya Kumari Shivakumar Kristina Lerman HILM 59 1 0 16 Feb 2025
InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion Zhaoyi Yan Zhijie Sang Yiming Zhang Yuhao Fu Baoyi He Qi Zhou Yining Di Chunlin Ji Shengyu Zhang Fei Wu MoMe LRM 64 1 0 06 Jan 2025
Multi-Level Optimal Transport for Universal Cross-Tokenizer Knowledge Distillation on Language Models Xiao Cui Mo Zhu Yulei Qin Liang Xie Wengang Zhou Yiming Li 91 4 0 19 Dec 2024
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 73 5 0 22 Oct 2024
Dual-Space Knowledge Distillation for Large Language Models Songming Zhang Xue Zhang Zengkui Sun Yufeng Chen Jinan Xu 50 5 0 25 Jun 2024
A Survey on Knowledge Distillation of Large Language Models Xiaohan Xu Ming Li Chongyang Tao Tao Shen Reynold Cheng Jinyang Li Can Xu Dacheng Tao Dinesh Manocha KELM VLM 44 102 0 20 Feb 2024
The Falcon Series of Open Language Models Ebtesam Almazrouei Hamza Alobeidli Abdulaziz Alshamsi Alessandro Cappelli Ruxandra-Aimée Cojocaru ... Quentin Malartic Daniele Mazzotta Badreddine Noune B. Pannier Guilherme Penedo AI4TS ALM 121 404 0 28 Nov 2023
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes Lokesh Nagalapatti Chun-Liang Li Chih-Kuan Yeh Hootan Nakhost Yasuhisa Fujii Alexander Ratner Ranjay Krishna Chen-Yu Lee Tomas Pfister ALM 220 502 0 03 May 2023
LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions Minghao Wu Abdul Waheed Chiyu Zhang Muhammad Abdul-Mageed Alham Fikri Aji ALM 135 119 0 27 Apr 2023
Sparks of Artificial General Intelligence: Early experiments with GPT-4 Sébastien Bubeck Varun Chandrasekaran Ronen Eldan J. Gehrke Eric Horvitz ... Scott M. Lundberg Harsha Nori Hamid Palangi Marco Tulio Ribeiro Yi Zhang ELM AI4MH AI4CE ALM 339 2,232 0 22 Mar 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 366 12,003 0 04 Mar 2022
KNOT: Knowledge Distillation using Optimal Transport for Solving NLP Tasks Rishabh Bhardwaj Tushar Vaidya Soujanya Poria OT FedML 65 7 0 06 Oct 2021
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 243 815 0 13 Sep 2019