Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective

1 February 2021

Papers citing "Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective"

34 / 34 papers shown

Title
ABKD: Pursuing a Proper Allocation of the Probability Mass in Knowledge Distillation via $α$ - $β$ -Divergence Guanghui Wang Zhiyong Yang Zihan Wang Shi Wang Qianqian Xu Qingming Huang 42 0 0 07 May 2025
Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data Yuting He Yiqiang Chen Xiaodong Yang H. Yu Yi-Hua Huang Yang Gu FedML 63 20 0 20 Apr 2025
YouLeQD: Decoding the Cognitive Complexity of Questions and Engagement in Online Educational Videos from Learners' Perspectives Nong Ming Sachin Sharma Jiho Noh AI4Ed 44 0 0 20 Jan 2025
Classroom-Inspired Multi-Mentor Distillation with Adaptive Learning Strategies Shalini Sarode Muhammad Saif Ullah Khan Tahira Shehzadi Didier Stricker Muhammad Zeshan Afzal 41 0 0 30 Sep 2024
Harmonizing knowledge Transfer in Neural Network with Unified Distillation Yaomin Huang Zaomin Yan Chaomin Shen Faming Fang Guixu Zhang 34 0 0 27 Sep 2024
Frequency-Guided Masking for Enhanced Vision Self-Supervised Learning Amin Karimi Monsefi Mengxi Zhou Nastaran Karimi Monsefi Ser-Nam Lim Wei-Lun Chao R. Ramnath 46 1 0 16 Sep 2024
Compact Language Models via Pruning and Knowledge Distillation Saurav Muralidharan Sharath Turuvekere Sreenivas Raviraj Joshi Marcin Chochowski M. Patwary M. Shoeybi Bryan Catanzaro Jan Kautz Pavlo Molchanov SyDa MQ 42 37 0 19 Jul 2024
Relational Representation Distillation Nikolaos Giakoumoglou Tania Stathaki 40 0 0 16 Jul 2024
Partner in Crime: Boosting Targeted Poisoning Attacks against Federated Learning Shihua Sun Shridatt Sugrim Angelos Stavrou Haining Wang AAML 63 1 0 13 Jul 2024
A Label is Worth a Thousand Images in Dataset Distillation Tian Qin Zhiwei Deng David Alvarez-Melis DD 94 10 0 15 Jun 2024
GIFT: Unlocking Full Potential of Labels in Distilled Dataset at Near-zero Cost Xinyi Shang Peng Sun Tao Lin 53 2 0 23 May 2024
MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities Kunxi Li Tianyu Zhan Kairui Fu Shengyu Zhang Kun Kuang Jiwei Li Zhou Zhao Fei Wu MoMe 24 0 0 20 Apr 2024
LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving Sicen Guo Zhiyuan Wu Qijun Chen Ioannis Pitas Rui Fan Rui Fan 40 1 0 13 Mar 2024
Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation Chengming Hu Haolun Wu Xuan Li Chen Ma Xi Chen Jun Yan Boyu Wang Xue Liu 35 3 0 22 Dec 2023
Student-friendly Knowledge Distillation Mengyang Yuan Bo Lang Fengnan Quan 20 17 0 18 May 2023
Do Not Blindly Imitate the Teacher: Using Perturbed Loss for Knowledge Distillation Rongzhi Zhang Jiaming Shen Tianqi Liu Jia-Ling Liu Michael Bendersky Marc Najork Chao Zhang 48 18 0 08 May 2023
Knowledge Distillation Under Ideal Joint Classifier Assumption Huayu Li Xiwen Chen G. Ditzler Janet Roveda Ao Li 18 1 0 19 Apr 2023
From Knowledge Distillation to Self-Knowledge Distillation: A Unified Approach with Normalized Loss and Customized Soft Labels Zhendong Yang Ailing Zeng Zhe Li Tianke Zhang Chun Yuan Yu Li 29 73 0 23 Mar 2023
Jaccard Metric Losses: Optimizing the Jaccard Index with Soft Labels Zifu Wang Xuefei Ning Matthew B. Blaschko VLM 30 12 0 11 Feb 2023
Rethinking Soft Label in Label Distribution Learning Perspective Seungbum Hong Jihun Yoon Bogyu Park Min-Kook Choi 31 0 0 31 Jan 2023
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images Yuan Yao Yuanhan Zhang Zhen-fei Yin Jiebo Luo Wanli Ouyang Xiaoshui Huang 3DPC 29 10 0 17 Dec 2022
D $^3$ ETR: Decoder Distillation for Detection Transformer Xiaokang Chen Jiahui Chen Yong-Jin Liu Gang Zeng 42 16 0 17 Nov 2022
Respecting Transfer Gap in Knowledge Distillation Yulei Niu Long Chen Chan Zhou Hanwang Zhang 26 23 0 23 Oct 2022
Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation Lirong Wu Yufei Huang Haitao Lin Zicheng Liu Tianyu Fan Stan Z. Li SSL 56 5 0 05 Oct 2022
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy Jingru Li Sheng Zhou Liangcheng Li Haishuai Wang Zhi Yu Jiajun Bu 34 14 0 29 Aug 2022
Rethinking Knowledge Distillation via Cross-Entropy Zhendong Yang Zhe Li Yuan Gong Tianke Zhang Shanshan Lao Chun Yuan Yu Li 30 14 0 22 Aug 2022
Boosting Multi-Label Image Classification with Complementary Parallel Self-Distillation Jiazhi Xu Sheng Huang Fengtao Zhou Luwen Huangfu D. Zeng Bo Liu 22 13 0 23 May 2022
Masked Generative Distillation Zhendong Yang Zhe Li Mingqi Shao Dachuan Shi Zehuan Yuan Chun Yuan FedML 32 168 0 03 May 2022
Cross-Image Relational Knowledge Distillation for Semantic Segmentation Chuanguang Yang Helong Zhou Zhulin An Xue Jiang Yong Xu Qian Zhang 37 169 0 14 Apr 2022
Knowledge Distillation as Efficient Pre-training: Faster Convergence, Higher Data-efficiency, and Better Transferability Ruifei He Shuyang Sun Jihan Yang Song Bai Xiaojuan Qi 34 36 0 10 Mar 2022
Optimizing for In-memory Deep Learning with Emerging Memory Technology Zhehui Wang Tao Luo Rick Siow Mong Goh Wei Zhang Weng-Fai Wong 18 1 0 01 Dec 2021
Prune Your Model Before Distill It Jinhyuk Park Albert No VLM 46 27 0 30 Sep 2021
R-Drop: Regularized Dropout for Neural Networks Xiaobo Liang Lijun Wu Juntao Li Yue Wang Qi Meng Tao Qin Wei Chen Hao Fei Tie-Yan Liu 47 424 0 28 Jun 2021
To Smooth or Not? When Label Smoothing Meets Noisy Labels Jiaheng Wei Hangyu Liu Tongliang Liu Gang Niu Masashi Sugiyama Yang Liu NoLa 32 69 0 08 Jun 2021