Why distillation helps: a statistical perspective

Why distillation helps: a statistical perspective

21 May 2020

Sashank J. Reddi

Sanjiv Kumar

ArXiv (abs)PDF HTML

Papers citing "Why distillation helps: a statistical perspective"

18 / 18 papers shown

Title
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Laws M. E. Ildiz Halil Alperen Gozeten Ege Onur Taga Marco Mondelli Samet Oymak 128 5 0 24 Oct 2024
Towards the Fundamental Limits of Knowledge Transfer over Finite Domains Qingyue Zhao Banghua Zhu 86 4 0 11 Oct 2023
Triplet Knowledge Distillation Xijun Wang Dongyang Liu Meina Kan Chunrui Han Zhongqin Wu Shiguang Shan 63 3 0 25 May 2023
Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR DongSeon Hwang K. Sim Yu Zhang Trevor Strohman 53 11 0 11 Oct 2022
Robust Active Distillation Cenk Baykal Khoa Trinh Fotis Iliopoulos Gaurav Menghani Erik Vee 77 11 0 03 Oct 2022
Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classification Quanshi Zhang Xu Cheng Yilan Chen Zhefan Rao 56 35 0 18 Aug 2022
Knowledge Distillation for Oriented Object Detection on Aerial Images Yicheng Xiao Junpeng Zhang ObjD 80 1 0 20 Jun 2022
Knowledge Distillation: Bad Models Can Be Good Role Models Gal Kaplun Eran Malach Preetum Nakkiran Shai Shalev-Shwartz FedML 58 15 0 28 Mar 2022
Towards Model Agnostic Federated Learning Using Knowledge Distillation A. Afonin Sai Praneeth Karimireddy FedML 87 45 0 28 Oct 2021
When in Doubt, Summon the Titans: Efficient Inference with Large Models A. S. Rawat Manzil Zaheer A. Menon Amr Ahmed Sanjiv Kumar 33 7 0 19 Oct 2021
S-LIME: Stabilized-LIME for Model Explanation Zhengze Zhou Giles Hooker Fei Wang FAtt 157 91 0 15 Jun 2021
Churn Reduction via Distillation Heinrich Jiang Harikrishna Narasimhan Dara Bahri Andrew Cotter Afshin Rostamizadeh 134 15 0 04 Jun 2021
Knowledge Distillation as Semiparametric Inference Tri Dao G. Kamath Vasilis Syrgkanis Lester W. Mackey 79 32 0 20 Apr 2021
Distilling Double Descent Andrew Cotter A. Menon Harikrishna Narasimhan A. S. Rawat Sashank J. Reddi Yichen Zhou 58 7 0 13 Feb 2021
Dual-Teacher++: Exploiting Intra-domain and Inter-domain Knowledge with Reliable Transfer for Cardiac Segmentation Kang Li Shujun Wang Lequan Yu Pheng-Ann Heng 111 29 0 07 Jan 2021
On Focal Loss for Class-Posterior Probability Estimation: A Theoretical Perspective Nontawat Charoenphakdee J. Vongkulbhisal Nuttapong Chairatanakul Masashi Sugiyama UQCV 46 26 0 18 Nov 2020
Hands-on Bayesian Neural Networks -- a Tutorial for Deep Learning Users Laurent Valentin Jospin Wray Buntine F. Boussaïd Hamid Laga Bennamoun OOD BDL UQCV 93 632 0 14 Jul 2020
Self-Distillation Amplifies Regularization in Hilbert Space H. Mobahi Mehrdad Farajtabar Peter L. Bartlett 85 235 0 13 Feb 2020

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.