v1v2 (latest)

Knowledge distillation: A good teacher is patient and consistent

9 June 2021

Papers citing "Knowledge distillation: A good teacher is patient and consistent"

50 / 199 papers shown

Title
Camera clustering for scalable stream-based active distillation Dani Manjah Davide Cacciarelli Christophe De Vleeschouwer Benoit Macq 91 1 0 16 Apr 2024
ReffAKD: Resource-efficient Autoencoder-based Knowledge Distillation Divyang Doshi Jung-Eun Kim 76 2 0 15 Apr 2024
Adversarial Robustness of Distilled and Pruned Deep Learning-based Wireless Classifiers N. Baishya B. R. Manoj AAML 39 0 0 11 Apr 2024
CodecLM: Aligning Language Models with Tailored Synthetic Data Zifeng Wang Chun-Liang Li Vincent Perot Long T. Le Jin Miao Zizhao Zhang Chen-Yu Lee Tomas Pfister SyDa ALM 73 21 0 08 Apr 2024
Tiny Machine Learning: Progress and Futures Ji Lin Ligeng Zhu Wei-Ming Chen Wei-Chen Wang Song Han 90 60 0 28 Mar 2024
Sentinel-Guided Zero-Shot Learning: A Collaborative Paradigm without Real Data Exposure Fan Wan Xingyu Miao Haoran Duan Jingjing Deng Rui Gao Yang Long VLM 86 5 0 14 Mar 2024
$V_kD:$ Improving Knowledge Distillation using Orthogonal Projections Roy Miles Ismail Elezi Jiankang Deng 114 10 0 10 Mar 2024
On the Effectiveness of Distillation in Mitigating Backdoors in Pre-trained Encoder Tingxu Han Shenghan Huang Ziqi Ding Weisong Sun Yebo Feng ... Hanwei Qian Cong Wu Quanjun Zhang Yang Liu Zhenyu Chen 54 8 0 06 Mar 2024
On Good Practices for Task-Specific Distillation of Large Pretrained Visual Models Juliette Marrie Michael Arbel Julien Mairal Diane Larlus VLM MQ 92 1 0 17 Feb 2024
Good Teachers Explain: Explanation-Enhanced Knowledge Distillation Amin Parchami-Araghi Moritz Bohle Sukrut Rao Bernt Schiele FAtt 61 4 0 05 Feb 2024
Contextualization Distillation from Large Language Model for Knowledge Graph Completion Dawei Li Zhen Tan Tianlong Chen Huan Liu KELM 107 13 0 28 Jan 2024
Model Compression Techniques in Biometrics Applications: A Survey Eduarda Caldeira Pedro C. Neto Marco Huber Naser Damer Ana F. Sequeira 119 11 0 18 Jan 2024
Bayes Conditional Distribution Estimation for Knowledge Distillation Based on Conditional Mutual Information Linfeng Ye Shayan Mohajer Hamidi Renhao Tan En-Hui Yang VLM 78 15 0 16 Jan 2024
An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation Md Arafat Sultan Aashka Trivedi Parul Awasthy Avirup Sil 79 0 0 12 Jan 2024
Source-Free Cross-Modal Knowledge Transfer by Unleashing the Potential of Task-Irrelevant Data Jinjin Zhu Yucheng Chen Lin Wang 86 2 0 10 Jan 2024
Data-Centric Foundation Models in Computational Healthcare: A Survey Yunkun Zhang Jin Gao Zheling Tan Lingfeng Zhou Kexin Ding Mu Zhou Shaoting Zhang Dequan Wang AI4CE 113 25 0 04 Jan 2024
Multi-modal Learning with Missing Modality in Predicting Axillary Lymph Node Metastasis Shichuan Zhang Sunyi Zheng Zhongyi Shui Honglin Li Lin Yang 49 5 0 03 Jan 2024
Compressing Deep Image Super-resolution Models Yuxuan Jiang Jakub Nawala Fan Zhang David Bull 96 7 0 31 Dec 2023
FerKD: Surgical Label Adaptation for Efficient Distillation Zhiqiang Shen 73 4 0 29 Dec 2023
Adaptive Depth Networks with Skippable Sub-Paths Woochul Kang 67 1 0 27 Dec 2023
StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation Shiu-hong Kao Jierun Chen S.-H. Gary Chan 73 0 0 20 Dec 2023
On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width Satoki Ishikawa Ryo Karakida 110 2 0 19 Dec 2023
Decoupled Knowledge with Ensemble Learning for Online Distillation Baitan Shao Ying Chen 44 0 0 18 Dec 2023
Mixed Distillation Helps Smaller Language Model Better Reasoning Chenglin Li Qianglong Chen Liangyue Li Wang Caiyu Yicheng Li Zhang Yin Yin Zhang LRM 79 15 0 17 Dec 2023
Let All be Whitened: Multi-teacher Distillation for Efficient Visual Retrieval Zhe Ma Jianfeng Dong Shouling Ji Zhenguang Liu Xuhong Zhang Zonghui Wang Sifeng He Feng Qian Xiaobo Zhang Lei Yang 78 6 0 15 Dec 2023
Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision Collin Burns Pavel Izmailov Jan Hendrik Kirchner Bowen Baker Leo Gao ... Adrien Ecoffet Manas Joglekar Jan Leike Ilya Sutskever Jeff Wu ELM 140 299 0 14 Dec 2023
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction Yi Guo Yiqian He Xiaoyang Li Haotong Qin Van Tung Pham Yang Zhang Shouda Liu 105 1 0 14 Dec 2023
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One Michael Ranzinger Greg Heinrich Jan Kautz Pavlo Molchanov VLM 167 50 0 10 Dec 2023
Beneath the Surface: Unveiling Harmful Memes with Multimodal Reasoning Distilled from Large Language Models Hongzhan Lin Ziyang Luo Jing Ma Long Chen 60 12 0 09 Dec 2023
Bootstrapping SparseFormers from Vision Foundation Models Ziteng Gao Zhan Tong Kevin Qinghong Lin Joya Chen Mike Zheng Shou 55 0 0 04 Dec 2023
MoEC: Mixture of Experts Implicit Neural Compression Jianchen Zhao Cheng-Ching Tseng Ming Lu Ruichuan An Xiaobao Wei He Sun Shanghang Zhang 81 3 0 03 Dec 2023
Initializing Models with Larger Ones Zhiqiu Xu Yanjie Chen Kirill Vishniakov Yida Yin Zhiqiang Shen Trevor Darrell Lingjie Liu Zhuang Liu 93 21 0 30 Nov 2023
MobileCLIP: Fast Image-Text Models through Multi-Modal Reinforced Training Pavan Kumar Anasosalu Vasu Hadi Pouransari Fartash Faghri Raviteja Vemulapalli Oncel Tuzel CLIP VLM 119 53 0 28 Nov 2023
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning Seonghak Kim Gyeongdo Ham Yucheol Cho Daeshik Kim 125 4 0 23 Nov 2023
Preference-Consistent Knowledge Distillation for Recommender System Zhangchi Zhu Wei Zhang 16 0 0 08 Nov 2023
MixtureGrowth: Growing Neural Networks by Recombining Learned Parameters Chau Pham Piotr Teterwak Soren Nelson Bryan A. Plummer 91 4 0 07 Nov 2023
Comparative Knowledge Distillation Alex Wilf Alex Tianyi Xu Paul Pu Liang A. Obolenskiy Daniel Fried Louis-Philippe Morency VLM 76 1 0 03 Nov 2023
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model Karsten Roth Lukas Thede Almut Sophia Koepke Oriol Vinyals Olivier J. Hénaff Zeynep Akata AAML 119 13 0 26 Oct 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 94 0 0 26 Oct 2023
Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models Florian Schmid Khaled Koutini Gerhard Widmer 47 11 0 24 Oct 2023
Data-Free Knowledge Distillation Using Adversarially Perturbed OpenGL Shader Images Logan Frank Jim Davis 77 1 0 20 Oct 2023
Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models Ruida Wang Wangchunshu Zhou Mrinmaya Sachan 95 32 0 20 Oct 2023
How a student becomes a teacher: learning and forgetting through Spectral methods Lorenzo Giambagli L. Buffoni Lorenzo Chicchi Duccio Fanelli 25 7 0 19 Oct 2023
What do larger image classifiers memorise? Michal Lukasik Vaishnavh Nagarajan A. S. Rawat A. Menon Sanjiv Kumar 100 5 0 09 Oct 2023
Talking Models: Distill Pre-trained Knowledge to Downstream Models via Interactive Communication Zhe Zhao Qingyun Liu Huan Gui Bang An Lichan Hong Ed H. Chi 77 1 0 04 Oct 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 83 1 0 20 Sep 2023
ICLEF: In-Context Learning with Expert Feedback for Explainable Style Transfer Arkadiy Saakyan Smaranda Muresan 91 4 0 15 Sep 2023
Self-Training and Multi-Task Learning for Limited Data: Evaluation Study on Object Detection Hoàng-Ân Lê Minh-Tan Pham 109 2 0 12 Sep 2023
Towards Comparable Knowledge Distillation in Semantic Image Segmentation Onno Niemann Christopher Vox Thorben Werner VLM 78 1 0 07 Sep 2023
EgoBlur: Responsible Innovation in Aria Nikhil Raina Guruprasad Somasundaram Kang Zheng Sagar Miglani Steve Saarinen ... Prince Gupta Mingfei Yan Richard Newcombe C. Ren Omkar M. Parkhi CVBM 46 9 0 24 Aug 2023