FitNets: Hints for Thin Deep Nets

19 December 2014

Adriana Romero

Nicolas Ballas

Samira Ebrahimi Kahou

Papers citing "FitNets: Hints for Thin Deep Nets"

50 / 725 papers shown

Title
Towards domain generalisation in ASR with elitist sampling and ensemble knowledge distillation Rehan Ahmad Md. Asif Jalal Muhammad Umar Farooq A. Ollerenshaw Thomas Hain 18 2 0 01 Mar 2023
Generic-to-Specific Distillation of Masked Autoencoders Wei Huang Zhiliang Peng Li Dong Furu Wei Jianbin Jiao QiXiang Ye 32 23 0 28 Feb 2023
Analyzing Populations of Neural Networks via Dynamical Model Embedding Jordan S. Cotler Kai Sheng Tai Felipe Hernández Blake Elias David Sussillo 22 4 0 27 Feb 2023
Graph-based Knowledge Distillation: A survey and experimental evaluation Jing Liu Tongya Zheng Guanzheng Zhang Qinfen Hao 35 8 0 27 Feb 2023
OccDepth: A Depth-Aware Method for 3D Semantic Scene Completion Ruihang Miao Weizhou Liu Ming-lei Chen Zheng Gong Weixin Xu Chen Hu Shuchang Zhou 35 81 0 27 Feb 2023
LightTS: Lightweight Time Series Classification with Adaptive Ensemble Distillation -- Extended Version David Campos Miao Zhang B. Yang Tung Kieu Chenjuan Guo Christian S. Jensen AI4TS 45 47 0 24 Feb 2023
Distilling Calibrated Student from an Uncalibrated Teacher Ishan Mishra Sethu Vamsi Krishna Deepak Mishra FedML 40 2 0 22 Feb 2023
Debiased Distillation by Transplanting the Last Layer Jiwoon Lee Jaeho Lee 23 3 0 22 Feb 2023
Two-in-one Knowledge Distillation for Efficient Facial Forgery Detection Chu Zhou Jiajun Huang Daochang Liu Chengbin Du Siqi Ma Surya Nepal Chang Xu 33 0 0 21 Feb 2023
HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers Chen Liang Haoming Jiang Zheng Li Xianfeng Tang Bin Yin Tuo Zhao VLM 29 24 0 19 Feb 2023
LEALLA: Learning Lightweight Language-agnostic Sentence Embeddings with Knowledge Distillation Zhuoyuan Mao Tetsuji Nakagawa FedML 19 19 0 16 Feb 2023
Take a Prior from Other Tasks for Severe Blur Removal Pei Wang Danna Xue Yu Zhu Jinqiu Sun Qingsen Yan Sung-eui Yoon Yanning Zhang 36 2 0 14 Feb 2023
Audio Representation Learning by Distilling Video as Privileged Information Amirhossein Hajavi Ali Etemad 21 4 0 06 Feb 2023
Rethinking Soft Label in Label Distribution Learning Perspective Seungbum Hong Jihun Yoon Bogyu Park Min-Kook Choi 31 0 0 31 Jan 2023
Supervision Complexity and its Role in Knowledge Distillation Hrayr Harutyunyan A. S. Rawat A. Menon Seungyeon Kim Surinder Kumar 32 12 0 28 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 55 121 0 17 Jan 2023
CrysGNN : Distilling pre-trained knowledge to enhance property prediction for crystalline materials Kishalay Das Bidisha Samanta Pawan Goyal Seung-Cheol Lee S. Bhattacharjee Niloy Ganguly 23 17 0 14 Jan 2023
InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers Leonid Boytsov Preksha Patel Vivek Sourabh Riddhi Nisar Sayan Kundu R. Ramanathan Eric Nyberg 34 19 0 08 Jan 2023
Explainability and Robustness of Deep Visual Classification Models Jindong Gu AAML 47 2 0 03 Jan 2023
TinyMIM: An Empirical Study of Distilling MIM Pre-trained Models Sucheng Ren Fangyun Wei Zheng-Wei Zhang Han Hu 42 35 0 03 Jan 2023
BD-KD: Balancing the Divergences for Online Knowledge Distillation Ibtihel Amara N. Sepahvand B. Meyer W. Gross J. Clark 32 2 0 25 Dec 2022
Exploring Content Relationships for Distilling Efficient GANs Lizhou You Mingbao Lin Tie Hu Rongrong Ji Rongrong Ji 49 3 0 21 Dec 2022
Training Lightweight Graph Convolutional Networks with Phase-field Models H. Sahbi 32 0 0 19 Dec 2022
Gait Recognition Using 3-D Human Body Shape Inference Haidong Zhu Zhao-Heng Zheng Ramkant Nevatia CVBM 3DH 36 23 0 18 Dec 2022
3D Point Cloud Pre-training with Knowledge Distillation from 2D Images Yuan Yao Yuanhan Zhang Zhen-fei Yin Jiebo Luo Wanli Ouyang Xiaoshui Huang 3DPC 29 10 0 17 Dec 2022
Autoencoders as Cross-Modal Teachers: Can Pretrained 2D Image Transformers Help 3D Representation Learning? Runpei Dong Zekun Qi Linfeng Zhang Junbo Zhang Jian‐Yuan Sun Zheng Ge Li Yi Kaisheng Ma ViT 3DPC 29 84 0 16 Dec 2022
Multi-scale Feature Imitation for Unsupervised Anomaly Localization Chao Hu Shengxin Lai 19 0 0 12 Dec 2022
LEAD: Liberal Feature-based Distillation for Dense Retrieval Hao Sun Xiao Liu Yeyun Gong Anlei Dong Jing Lu Yan Zhang Linjun Yang Rangan Majumder Nan Duan 67 2 0 10 Dec 2022
Deep Incubation: Training Large Models by Divide-and-Conquering Zanlin Ni Yulin Wang Jiangwei Yu Haojun Jiang Yu Cao Gao Huang VLM 20 11 0 08 Dec 2022
Occlusion-Robust FAU Recognition by Mining Latent Space of Masked Autoencoders Minyang Jiang Yongwei Wang Martin J. McKeown Jane Wang CVBM 20 2 0 08 Dec 2022
BEV-LGKD: A Unified LiDAR-Guided Knowledge Distillation Framework for BEV 3D Object Detection Jianing Li Ming Lu Jiaming Liu Yandong Guo Li Du Shanghang Zhang 44 6 0 01 Dec 2022
Hint-dynamic Knowledge Distillation Yiyang Liu Chenxin Li Xiaotong Tu Xinghao Ding Yue Huang 14 1 0 30 Nov 2022
Attention-Based Depth Distillation with 3D-Aware Positional Encoding for Monocular 3D Object Detection Zizhang Wu Yunzhe Wu Jian Pu Xianzhi Li Xiaoquan Wang 30 14 0 30 Nov 2022
Curriculum Temperature for Knowledge Distillation Zheng Li Xiang Li Lingfeng Yang Borui Zhao Renjie Song Lei Luo Jun Yu Li Jian Yang 33 134 0 29 Nov 2022
Rethinking Implicit Neural Representations for Vision Learners Yiran Song Qianyu Zhou Lizhuang Ma 24 7 0 22 Nov 2022
Accelerating Diffusion Sampling with Classifier-based Feature Distillation Wujie Sun Defang Chen Can Wang Deshi Ye Yan Feng Chun-Yen Chen 35 16 0 22 Nov 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 40 6 0 17 Nov 2022
D $^3$ ETR: Decoder Distillation for Detection Transformer Xiaokang Chen Jiahui Chen Yong-Jin Liu Gang Zeng 42 16 0 17 Nov 2022
Yield Evaluation of Citrus Fruits based on the YoloV5 compressed by Knowledge Distillation Yuqi Li Yuting He Yihang Zhou Zirui Gong Ren-Jun Huang 21 0 0 16 Nov 2022
Structured Knowledge Distillation Towards Efficient and Compact Multi-View 3D Detection Linfeng Zhang Yukang Shi Hung-Shuo Tai Zhipeng Zhang Yuan He Ke Wang Kaisheng Ma 28 2 0 14 Nov 2022
Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection Yu Hong Hang Dai Yong Ding 3DPC 32 71 0 14 Nov 2022
Partial Binarization of Neural Networks for Budget-Aware Efficient Learning Udbhav Bamba Neeraj Anand Saksham Aggarwal Dilip K Prasad D. K. Gupta MQ 26 0 0 12 Nov 2022
Multimodal Dyadic Impression Recognition via Listener Adaptive Cross-Domain Fusion Yuanchao Li P. Bell Catherine Lai 33 3 0 09 Nov 2022
Hilbert Distillation for Cross-Dimensionality Networks Dian Qin Haishuai Wang Zhe Liu Hongjia Xu Sheng Zhou Jiajun Bu 23 4 0 08 Nov 2022
Distilling Representations from GAN Generator via Squeeze and Span Yu Yang Xiaotian Cheng Chang-rui Liu Hakan Bilen Xiang Ji GAN 33 0 0 06 Nov 2022
Eliciting Knowledge from Large Pre-Trained Models for Unsupervised Knowledge-Grounded Conversation Yanyang Li Jianqiao Zhao M. Lyu Liwei Wang 24 15 0 03 Nov 2022
MPCFormer: fast, performant and private Transformer inference with MPC Dacheng Li Rulin Shao Hongyi Wang Han Guo Eric P. Xing Haotong Zhang 21 80 0 02 Nov 2022
Gradient Knowledge Distillation for Pre-trained Language Models Lean Wang Lei Li Xu Sun VLM 28 5 0 02 Nov 2022
Maximum Likelihood Distillation for Robust Modulation Classification Javier Maroto Gérôme Bovet P. Frossard AAML 18 5 0 01 Nov 2022
SADT: Combining Sharpness-Aware Minimization with Self-Distillation for Improved Model Generalization Masud An Nur Islam Fahim Jani Boutellier 42 0 0 01 Nov 2022