Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

1 April 2021

Papers citing "Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study"

48 / 48 papers shown

Title
MaxSup: Overcoming Representation Collapse in Label Smoothing Yuxuan Zhou Heng Li Zhi-Qi Cheng Xudong Yan Yifei Dong Mario Fritz Margret Keuper 160 0 0 18 Feb 2025
Overcoming Language Priors for Visual Question Answering Based on Knowledge Distillation Daowan Peng Wei Wei 485 1 0 10 Jan 2025
Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation Lirong Wu Yunfan Liu Haitao Lin Yufei Huang Stan Z. Li 93 1 0 20 Jul 2024
Exploring Dark Knowledge under Various Teacher Capacities and Addressing Capacity Mismatch Wen-Shu Fan Xin-Chun Li Bowen Tao 84 2 0 21 May 2024
Enhancing Suicide Risk Detection on Social Media through Semi-Supervised Deep Label Smoothing Matthew Squires Xiaohui Tao Soman Elangovan Rajendra Acharya R. Gururajan Haoran Xie Xujuan Zhou AI4MH 69 2 0 09 May 2024
MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities Kunxi Li Tianyu Zhan Kairui Fu Shengyu Zhang Kun Kuang Jiwei Li Zhou Zhao Leilei Gan MoMe 65 3 0 20 Apr 2024
Revisiting Confidence Estimation: Towards Reliable Failure Prediction Fei Zhu Xu-Yao Zhang Zhen Cheng Cheng-Lin Liu UQCV 103 12 0 05 Mar 2024
Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learning Yuhang Zheng Zhen Wang Long Chen 68 2 0 28 Jan 2024
FerKD: Surgical Label Adaptation for Efficient Distillation Zhiqiang Shen 81 4 0 29 Dec 2023
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS Zhiwen Fan Kevin Wang Kairun Wen Zehao Zhu Dejia Xu Zhangyang Wang 3DGS 159 222 0 28 Nov 2023
Towards Better Query Classification with Multi-Expert Knowledge Condensation in JD Ads Search Hai-Jian Ke Ming Pang Zheng Fang Xue Jiang Xi-Wei Zhao Changping Peng Zhangang Lin Jinghe Hu Jingping Shao 124 0 0 02 Aug 2023
DOT: A Distillation-Oriented Trainer Borui Zhao Quan Cui Renjie Song Jiajun Liang 62 7 0 17 Jul 2023
Open-Set RF Fingerprinting via Improved Prototype Learning Weidong Wang H. Liao Lu Gan 55 0 0 24 Jun 2023
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation Chuanguang Yang Xinqiang Yu Zhulin An Yongjun Xu VLM OffRL 206 27 0 19 Jun 2023
Heterogeneous Continual Learning Divyam Madaan Hongxu Yin Wonmin Byeon Jan Kautz Pavlo Molchanov CLL 76 5 0 14 Jun 2023
On the Impact of Knowledge Distillation for Model Interpretability Hyeongrok Han Siwon Kim Hyun-Soo Choi Sungroh Yoon 68 5 0 25 May 2023
Student-friendly Knowledge Distillation Mengyang Yuan Bo Lang Fengnan Quan 92 21 0 18 May 2023
LABO: Towards Learning Optimal Label Regularization via Bi-level Optimization Peng Lu Ahmad Rashid I. Kobyzev Mehdi Rezagholizadeh Philippe Langlais 61 0 0 08 May 2023
Re-thinking Model Inversion Attacks Against Deep Neural Networks Ngoc-Bao Nguyen Keshigeyan Chandrasegaran Milad Abdollahzadeh Ngai-Man Cheung 93 44 0 04 Apr 2023
Out of Thin Air: Exploring Data-Free Adversarial Robustness Distillation Yuzheng Wang Zhaoyu Chen Dingkang Yang Pinxue Guo Kaixun Jiang Wenqiang Zhang Lizhe Qi AAML 67 6 0 21 Mar 2023
Reinforce Data, Multiply Impact: Improved Model Accuracy and Robustness with Dataset Reinforcement Fartash Faghri Hadi Pouransari Sachin Mehta Mehrdad Farajtabar Ali Farhadi Mohammad Rastegari Oncel Tuzel 88 9 0 15 Mar 2023
Rethinking Confidence Calibration for Failure Prediction Fei Zhu Zhen Cheng Xu-Yao Zhang Cheng-Lin Liu UQCV 95 41 0 06 Mar 2023
Rethinking Soft Label in Label Distribution Learning Perspective Seungbum Hong Jihun Yoon Bogyu Park Min-Kook Choi 72 0 0 31 Jan 2023
$Knowledge Distillation $\approx$ Label Smoothing: Fact or Fallacy?$ Knowledge Distillation $\approx$ Label Smoothing: Fact or Fallacy? Md Arafat Sultan 86 2 0 30 Jan 2023
Understanding the Role of Mixup in Knowledge Distillation: An Empirical Study Hongjun Choi Eunyeong Jeon Ankita Shukla Pavan Turaga 65 8 0 08 Nov 2022
Symbolic Distillation for Learned TCP Congestion Control S. Sharan Wenqing Zheng Kuo-Feng Hsu Jiarong Xing Ang Chen Zhangyang Wang 94 5 0 24 Oct 2022
Respecting Transfer Gap in Knowledge Distillation Yulei Niu Long Chen Chan Zhou Hanwang Zhang 104 26 0 23 Oct 2022
Asymmetric Temperature Scaling Makes Larger Networks Teach Well Again Xin-Chun Li Wenxuan Fan Shaoming Song Yinchuan Li Bingshuai Li Yunfeng Shao De-Chuan Zhan 123 31 0 10 Oct 2022
Preserving Fine-Grain Feature Information in Classification via Entropic Regularization Raphael Baena Lucas Drumetz Vincent Gripon 65 3 0 07 Aug 2022
PEA: Improving the Performance of ReLU Networks for Free by Using Progressive Ensemble Activations Á. Utasi 49 0 0 28 Jul 2022
TinyViT: Fast Pretraining Distillation for Small Vision Transformers Kan Wu Jinnian Zhang Houwen Peng Mengchen Liu Bin Xiao Jianlong Fu Lu Yuan ViT 87 267 0 21 Jul 2022
Knowledge Condensation Distillation Chenxin Li Mingbao Lin Zhiyuan Ding Nie Lin Yihong Zhuang Yue Huang Xinghao Ding Liujuan Cao 88 28 0 12 Jul 2022
PrUE: Distilling Knowledge from Sparse Teacher Networks Shaopu Wang Xiaojun Chen Mengzhen Kou Jinqiao Shi 125 2 0 03 Jul 2022
Improving the Robustness and Generalization of Deep Neural Network with Confidence Threshold Reduction Xiangyuan Yang Jie Lin Hanlin Zhang Xinyu Yang Peng Zhao AAML OOD 72 1 0 02 Jun 2022
Knowledge Distillation from A Stronger Teacher Tao Huang Shan You Fei Wang Chao Qian Chang Xu 119 258 0 21 May 2022
Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generation Xingning Dong Tian Gan Xuemeng Song Jianlong Wu Yuan Cheng Liqiang Nie 118 96 0 18 Mar 2022
Meta Knowledge Distillation Jihao Liu Boxiao Liu Hongsheng Li Yu Liu 85 26 0 16 Feb 2022
Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space Arnav Chavan Zhiqiang Shen Zhuang Liu Zechun Liu Kwang-Ting Cheng Eric P. Xing ViT 102 72 0 03 Jan 2022
Data-Free Neural Architecture Search via Recursive Label Calibration Zechun Liu Zhiqiang Shen Yun Long Eric P. Xing Kwang-Ting Cheng Chas Leichner 43 7 0 03 Dec 2021
A Fast Knowledge Distillation Framework for Visual Recognition Zhiqiang Shen Eric P. Xing VLM 112 50 0 02 Dec 2021
Sliced Recursive Transformer Zhiqiang Shen Zechun Liu Eric P. Xing ViT 61 27 0 09 Nov 2021
Network Augmentation for Tiny Deep Learning Han Cai Chuang Gan Ji Lin Song Han 133 30 0 17 Oct 2021
Class-Distribution-Aware Calibration for Long-Tailed Visual Recognition Mobarakol Islam Lalithkumar Seenivasan Hongliang Ren Ben Glocker 72 19 0 11 Sep 2021
Knowledge Inheritance for Pre-trained Language Models Yujia Qin Yankai Lin Jing Yi Jiajie Zhang Xu Han ... Yusheng Su Zhiyuan Liu Peng Li Maosong Sun Jie Zhou VLM 85 50 0 28 May 2021
Spending Your Winning Lottery Better After Drawing It Ajay Jaiswal Haoyu Ma Tianlong Chen Ying Ding Zhangyang Wang 68 6 0 08 Jan 2021
MEAL V2: Boosting Vanilla ResNet-50 to 80%+ Top-1 Accuracy on ImageNet without Tricks Zhiqiang Shen Marios Savvides 95 63 0 17 Sep 2020
Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning Zhiqiang Shen Zechun Liu Zhuang Liu Marios Savvides Trevor Darrell Eric P. Xing OCL SSL 104 103 0 11 Mar 2020
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions Zechun Liu Zhiqiang Shen Marios Savvides Kwang-Ting Cheng MQ 220 355 0 07 Mar 2020