Simple Unsupervised Knowledge Distillation With Space Similarity

20 September 2024

Papers citing "Simple Unsupervised Knowledge Distillation With Space Similarity"

35 / 35 papers shown

Title
Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages Shivanshu Gupta Yoshitomo Matsubara Ankita N. Chadha Alessandro Moschitti 64 2 0 25 May 2023
Attention Distillation: self-supervised vision transformer students need more guidance Kai Wang Fei Yang Joost van de Weijer ViT 44 18 0 03 Oct 2022
SimReg: Regression as a Simple Yet Effective Tool for Self-supervised Knowledge Distillation K. Navaneet Soroush Abbasi Koohpayegani Ajinkya Tejankar Hamed Pirsiavash 42 20 0 13 Jan 2022
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 611 6,029 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 146 1,857 0 05 Apr 2021
SEED: Self-supervised Distillation For Visual Representation Zhiyuan Fang Jianfeng Wang Lijuan Wang Lei Zhang Yezhou Yang Zicheng Liu SSL 293 191 0 12 Jan 2021
Wasserstein Contrastive Representation Distillation Liqun Chen Dong Wang Zhe Gan Jingjing Liu Ricardo Henao Lawrence Carin 70 95 0 15 Dec 2020
How Well Do Self-Supervised Models Transfer? Linus Ericsson Henry Gouk Timothy M. Hospedales SSL 99 278 0 26 Nov 2020
CompRess: Self-Supervised Learning by Compressing Representations Soroush Abbasi Koohpayegani Ajinkya Tejankar Hamed Pirsiavash SSL 63 90 0 28 Oct 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 543 40,739 0 22 Oct 2020
Measuring Robustness to Natural Distribution Shifts in Image Classification Rohan Taori Achal Dave Vaishaal Shankar Nicholas Carlini Benjamin Recht Ludwig Schmidt OOD 108 546 0 01 Jul 2020
Knowledge Distillation Meets Self-Supervision Guodong Xu Ziwei Liu Xiaoxiao Li Chen Change Loy FedML 69 283 0 12 Jun 2020
Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation Nils Reimers Iryna Gurevych 94 1,021 0 21 Apr 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 452 3,422 0 09 Mar 2020
Self-Supervised Learning of Pretext-Invariant Representations Ishan Misra Laurens van der Maaten SSL VLM 92 1,451 0 04 Dec 2019
Contrastive Representation Distillation Yonglong Tian Dilip Krishnan Phillip Isola 141 1,045 0 23 Oct 2019
On the Efficacy of Knowledge Distillation Ligang He Rui Mao 92 605 0 03 Oct 2019
Similarity-Preserving Knowledge Distillation Frederick Tung Greg Mori 113 973 0 23 Jul 2019
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift Yaniv Ovadia Emily Fertig Jie Jessie Ren Zachary Nado D. Sculley Sebastian Nowozin Joshua V. Dillon Balaji Lakshminarayanan Jasper Snoek UQCV 159 1,688 0 06 Jun 2019
Learning Robust Global Representations by Penalizing Local Predictive Power Haohan Wang Songwei Ge Eric Xing Zachary Chase Lipton OOD 109 957 0 29 May 2019
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Y. Yoo OOD 604 4,766 0 13 May 2019
Relational Knowledge Distillation Wonpyo Park Dongju Kim Yan Lu Minsu Cho 63 1,405 0 10 Apr 2019
Correlation Congruence for Knowledge Distillation Baoyun Peng Xiao Jin Jiaheng Liu Shunfeng Zhou Yichao Wu Yu Liu Dongsheng Li Zhaoning Zhang 86 510 0 03 Apr 2019
Benchmarking Neural Network Robustness to Common Corruptions and Perturbations Dan Hendrycks Thomas G. Dietterich OOD VLM 158 3,423 0 28 Mar 2019
Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey Longlong Jing Yingli Tian SSL 108 1,697 0 16 Feb 2019
Deep Mutual Learning Ying Zhang Tao Xiang Timothy M. Hospedales Huchuan Lu FedML 143 1,650 0 01 Jun 2017
Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer Sergey Zagoruyko N. Komodakis 113 2,569 0 12 Dec 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 342 10,172 0 16 Mar 2016
Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks Shaoqing Ren Kaiming He Ross B. Girshick Jian Sun AIMat ObjD 478 62,122 0 04 Jun 2015
Unsupervised Visual Representation Learning by Context Prediction Carl Doersch Abhinav Gupta Alexei A. Efros DRL SSL 164 2,782 0 19 May 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 432 43,234 0 11 Feb 2015
FitNets: Hints for Thin Deep Nets Adriana Romero Nicolas Ballas Samira Ebrahimi Kahou Antoine Chassang C. Gatta Yoshua Bengio FedML 286 3,870 0 19 Dec 2014
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 381 43,524 0 01 May 2014
Network In Network Min Lin Qiang Chen Shuicheng Yan 283 6,274 0 16 Dec 2013
Describing Textures in the Wild Mircea Cimpoi Subhransu Maji Iasonas Kokkinos S. Mohamed Andrea Vedaldi 3DV 106 2,661 0 14 Nov 2013