Improved Knowledge Distillation via Teacher Assistant

9 February 2019

Papers citing "Improved Knowledge Distillation via Teacher Assistant"

50 / 166 papers shown

Title
Catch-Up Distillation: You Only Need to Train Once for Accelerating Sampling Shitong Shao Xu Dai Shouyi Yin Lujun Li Huanran Chen Yang Hu 24 17 0 18 May 2023
Boost Vision Transformer with GPU-Friendly Sparsity and Quantization Chong Yu Tao Chen Zhongxue Gan Jiayuan Fan MQ ViT 30 23 0 18 May 2023
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation Yuxin Ren Zi-Qi Zhong Xingjian Shi Yi Zhu Chun Yuan Mu Li 24 7 0 16 May 2023
Analyzing Compression Techniques for Computer Vision Maniratnam Mandal Imran Khan 27 1 0 14 May 2023
A Systematic Study of Knowledge Distillation for Natural Language Generation with Pseudo-Target Training Nitay Calderon Subhabrata Mukherjee Roi Reichart Amir Kantor 41 17 0 03 May 2023
SoK: Pragmatic Assessment of Machine Learning for Network Intrusion Detection Giovanni Apruzzese Pavel Laskov J. Schneider 44 25 0 30 Apr 2023
LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks Di Hong Jiangrong Shen Yu Qi Yueming Wang 25 5 0 17 Apr 2023
Distilling Token-Pruned Pose Transformer for 2D Human Pose Estimation Feixiang Ren ViT 21 2 0 12 Apr 2023
Grouped Knowledge Distillation for Deep Face Recognition Weisong Zhao Xiangyu Zhu Kaiwen Guo Xiaoyu Zhang Zhen Lei CVBM 23 6 0 10 Apr 2023
MiniRBT: A Two-stage Distilled Small Chinese Pre-trained Model Xin Yao Ziqing Yang Yiming Cui Shijin Wang 23 3 0 03 Apr 2023
DIME-FM: DIstilling Multimodal and Efficient Foundation Models Ximeng Sun Pengchuan Zhang Peizhao Zhang Hardik Shah Kate Saenko Xide Xia VLM 25 20 0 31 Mar 2023
DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving Ju He Zhi-Qi Cheng Chenyang Li Wangmeng Xiang Binghui Chen Bin Luo Yifeng Geng Xuansong Xie AI4CE 31 20 0 30 Mar 2023
Decoupled Multimodal Distilling for Emotion Recognition Yong Li Yuan-Zheng Wang Zhen Cui 21 73 0 24 Mar 2023
MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation Vitaliy Kinakh M. Drozdova Slava Voloshynovskiy 40 1 0 21 Mar 2023
Towards a Smaller Student: Capacity Dynamic Distillation for Efficient Image Retrieval Yi Xie Huaidong Zhang Xuemiao Xu Jianqing Zhu Shengfeng He VLM 21 13 0 16 Mar 2023
Focus on Your Target: A Dual Teacher-Student Framework for Domain-adaptive Semantic Segmentation Xinyue Huo Lingxi Xie Wen-gang Zhou Houqiang Li Qi Tian 29 8 0 16 Mar 2023
Distillation from Heterogeneous Models for Top-K Recommendation SeongKu Kang Wonbin Kweon Dongha Lee Jianxun Lian Xing Xie Hwanjo Yu VLM 35 21 0 02 Mar 2023
Practical Knowledge Distillation: Using DNNs to Beat DNNs Chungman Lee Pavlos Anastasios Apostolopulos Igor L. Markov FedML 27 1 0 23 Feb 2023
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 27 24 0 19 Feb 2023
Improved Knowledge Distillation for Pre-trained Language Models via Knowledge Selection Chenglong Wang Yi Lu Yongyu Mu Yimin Hu Tong Xiao Jingbo Zhu 34 8 0 01 Feb 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 30 12 0 28 Jan 2023
Tailor: Altering Skip Connections for Resource-Efficient Inference Olivia Weng Gabriel Marcano Vladimir Loncar Alireza Khodamoradi Nojan Sheybani Andres Meza F. Koushanfar K. Denolf Javier Mauricio Duarte Ryan Kastner 46 11 0 18 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 53 121 0 17 Jan 2023
Interaction-Aware Trajectory Planning for Autonomous Vehicles with Analytic Integration of Neural Networks into Model Predictive Control Piyush Gupta David Isele Donggun Lee S. Bae 39 19 0 13 Jan 2023
BD-KD: Balancing the Divergences for Online Knowledge Distillation Ibtihel Amara N. Sepahvand B. Meyer W. Gross J. Clark 29 2 0 25 Dec 2022
CAMeMBERT: Cascading Assistant-Mediated Multilingual BERT Dan DeGenaro Jugal Kalita 32 0 0 22 Dec 2022
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 67 2 0 10 Dec 2022
Responsible Active Learning via Human-in-the-loop Peer Study Yu Cao Jingya Wang Baosheng Yu Dacheng Tao 25 0 0 24 Nov 2022
Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans? Neeraj Varshney Man Luo Chitta Baral RALM 21 11 0 23 Nov 2022
AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation Hyungmin Kim Sungho Suh Sunghyun Baek Daehwan Kim Daun Jeong Hansang Cho Junmo Kim 27 5 0 20 Nov 2022
Yield Evaluation of Citrus Fruits based on the YoloV5 compressed by Knowledge Distillation Yuqi Li Yuting He Yihang Zhou Zirui Gong Ren-Jun Huang 21 0 0 16 Nov 2022
AltCLIP: Altering the Language Encoder in CLIP for Extended Language Capabilities Zhongzhi Chen Guangyi Liu Bo-Wen Zhang Fulong Ye Qinghong Yang Ledell Yu Wu VLM 37 80 0 12 Nov 2022
Teacher-Student Architecture for Knowledge Learning: A Survey Chengming Hu Xuan Li Dan Liu Xi Chen Ju Wang Xue Liu 20 35 0 28 Oct 2022
Improved Feature Distillation via Projector Ensemble Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Zi Huang 37 37 0 27 Oct 2022
Exploiting Features and Logits in Heterogeneous Federated Learning Yun-Hin Chan Edith C.H. Ngai FedML 32 2 0 27 Oct 2022
Respecting Transfer Gap in Knowledge Distillation Yulei Niu Long Chen Chan Zhou Hanwang Zhang 26 23 0 23 Oct 2022
Similarity of Neural Architectures using Adversarial Attack Transferability Jaehui Hwang Dongyoon Han Byeongho Heo Song Park Sanghyuk Chun Jong-Seok Lee AAML 32 1 0 20 Oct 2022
Few-Shot Learning of Compact Models via Task-Specific Meta Distillation Yong Wu Shekhor Chanda M. Hosseinzadeh Zhi Liu Yang Wang VLM 29 7 0 18 Oct 2022
Approximating Continuous Convolutions for Deep Network Compression Theo W. Costain V. Prisacariu 36 0 0 17 Oct 2022
Knowledge Distillation Transfer Sets and their Impact on Downstream NLU Tasks Charith Peris Lizhen Tan Thomas Gueudré Turan Gojayev Vivi Wei Gokmen Oz 28 4 0 10 Oct 2022
Stimulative Training of Residual Networks: A Social Psychology Perspective of Loafing Peng Ye Shengji Tang Baopu Li Tao Chen Wanli Ouyang 31 13 0 09 Oct 2022
Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation Rahul Mishra Hari Prabhat Gupta 40 8 0 30 Sep 2022
Slimmable Networks for Contrastive Self-supervised Learning Shuai Zhao Xiaohan Wang Linchao Zhu Yi Yang 35 1 0 30 Sep 2022
PROD: Progressive Distillation for Dense Retrieval Zhenghao Lin Yeyun Gong Xiao Liu Hang Zhang Chen Lin ... Jian Jiao Jing Lu Daxin Jiang Rangan Majumder Nan Duan 51 27 0 27 Sep 2022
Multi-stage Distillation Framework for Cross-Lingual Semantic Similarity Matching Kunbo Ding Weijie Liu Yuejian Fang Zhe Zhao Qi Ju Xuefeng Yang 23 1 0 13 Sep 2022
Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy Jingru Li Sheng Zhou Liangcheng Li Haishuai Wang Zhi Yu Jiajun Bu 34 14 0 29 Aug 2022
Dense Depth Distillation with Out-of-Distribution Simulated Images Junjie Hu Chenyou Fan Mete Ozay Hualie Jiang Tin Lun Lam 24 4 0 26 Aug 2022
Masked Autoencoders Enable Efficient Knowledge Distillers Yutong Bai Zeyu Wang Junfei Xiao Chen Wei Huiyu Wang Alan Yuille Yuyin Zhou Cihang Xie CLL 32 39 0 25 Aug 2022
Task-Balanced Distillation for Object Detection Ruining Tang Zhen-yu Liu Yangguang Li Yiguo Song Hui Liu Qide Wang Jing Shao Guifang Duan Jianrong Tan 26 20 0 05 Aug 2022
Online Knowledge Distillation via Mutual Contrastive Learning for Visual Recognition Chuanguang Yang Zhulin An Helong Zhou Fuzhen Zhuang Yongjun Xu Qian Zhang 41 50 0 23 Jul 2022