Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

12 December 2016

Papers citing "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer"

50 / 1,157 papers shown

Title
MoMA: Momentum Contrastive Learning with Multi-head Attention-based Knowledge Distillation for Histopathology Image Analysis T. Vuong J. T. Kwak 41 6 0 31 Aug 2023
Adversarial Finetuning with Latent Representation Constraint to Mitigate Accuracy-Robustness Tradeoff Satoshi Suzuki Shin'ya Yamaguchi Shoichiro Takeda Sekitoshi Kanai Naoki Makishima Atsushi Ando Ryo Masumura AAML 34 5 0 31 Aug 2023
SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation Changze Lv Tianlong Li Jianhan Xu Chenxi Gu Zixuan Ling Cenyuan Zhang Xiaoqing Zheng Xuanjing Huang 42 8 0 29 Aug 2023
SynthDistill: Face Recognition with Knowledge Distillation from Synthetic Data Hatef Otroshi Anjith George S´ebastien Marcel 35 11 0 28 Aug 2023
Distilled GPT for Source Code Summarization Chia-Yi Su Collin McMillan 30 36 0 28 Aug 2023
Machine Unlearning Methodology base on Stochastic Teacher Network Xulong Zhang Jianzong Wang Ning Cheng Yifu Sun Chuanyao Zhang Jing Xiao MU 29 4 0 28 Aug 2023
Computation-efficient Deep Learning for Computer Vision: A Survey Yulin Wang Yizeng Han Chaofei Wang Shiji Song Qi Tian Gao Huang VLM 41 20 0 27 Aug 2023
Universal Graph Continual Learning Thanh Duc Hoang Do Viet Tung Duy-Hung Nguyen Bao-Sinh Nguyen Huy Hoang Nguyen Hung Le CLL 31 3 0 27 Aug 2023
REFT: Resource-Efficient Federated Training Framework for Heterogeneous and Resource-Constrained Environments Humaid Ahmed Desai Amr B. Hilal Hoda Eldardiry 29 0 0 25 Aug 2023
Fine-tuning can cripple your foundation model; preserving features may be the solution Jishnu Mukhoti Y. Gal Philip Torr P. Dokania CLL 43 32 0 25 Aug 2023
DLIP: Distilling Language-Image Pre-training Huafeng Kuang Jie Wu Xiawu Zheng Ming Li Xuefeng Xiao Rui Wang Min Zheng Rongrong Ji VLM 44 4 0 24 Aug 2023
SG-Former: Self-guided Transformer with Evolving Token Reallocation Sucheng Ren Xingyi Yang Songhua Liu Xinchao Wang ViT 27 41 0 23 Aug 2023
DR-Tune: Improving Fine-tuning of Pretrained Visual Models by Distribution Regularization with Semantic Calibration Nana Zhou Jiaxin Chen Di Huang 27 4 0 23 Aug 2023
Efficient View Synthesis with Neural Radiance Distribution Field Yushuang Wu Xiao Li Jinglu Wang Xiaoguang Han Shuguang Cui Yan Lu 23 1 0 22 Aug 2023
Diffusion Model as Representation Learner Xingyi Yang Xinchao Wang DiffM 37 55 0 21 Aug 2023
Representation Disparity-aware Distillation for 3D Object Detection Yanjing Li Sheng Xu Mingbao Lin Jihao Yin Baochang Zhang Xianbin Cao 22 3 0 20 Aug 2023
CCFace: Classification Consistency for Low-Resolution Face Recognition Mohammad Saeed Ebrahimi Saadabadi Sahar Rahimi Malakshan Hossein Kashiani Nasser M. Nasrabadi CVBM 33 4 0 18 Aug 2023
Learning Lightweight Object Detectors via Multi-Teacher Progressive Distillation Shengcao Cao Mengtian Li James Hays Deva Ramanan Yu-xiong Wang Liangyan Gui VLM 26 11 0 17 Aug 2023
SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations Zhiming Wang Lin Gu Feng Lu 32 0 0 17 Aug 2023
Radio2Text: Streaming Speech Recognition Using mmWave Radio Signals Running Zhao Jiang-Tao Luca Yu Haiying Zhao Edith C.H. Ngai 32 4 0 16 Aug 2023
Revisiting Vision Transformer from the View of Path Ensemble Shuning Chang Pichao Wang Haowen Luo Fan Wang Mike Zheng Shou ViT 40 3 0 12 Aug 2023
Multi-Label Knowledge Distillation Penghui Yang Ming-Kun Xie Chen-Chen Zong Lei Feng Gang Niu Masashi Sugiyama Sheng-Jun Huang 41 10 0 12 Aug 2023
Foreground Object Search by Distilling Composite Image Feature Bo Zhang Jiacheng Sui Li Niu 33 5 0 09 Aug 2023
Data-Free Model Extraction Attacks in the Context of Object Detection Harshit Shah G. Aravindhan Pavan Kulkarni Yuvaraj Govidarajulu Manojkumar Somabhai Parmar MIACV AAML 44 3 0 09 Aug 2023
Prune Spatio-temporal Tokens by Semantic-aware Temporal Accumulation Shuangrui Ding Peisen Zhao Xiaopeng Zhang Rui Qian H. Xiong Qi Tian ViT 29 16 0 08 Aug 2023
Teacher-Student Architecture for Knowledge Distillation: A Survey Chengming Hu Xuan Li Danyang Liu Haolun Wu Xi Chen Ju Wang Xue Liu 26 16 0 08 Aug 2023
AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation Amir M. Mansourian Rozhan Ahmadi S. Kasaei 44 2 0 08 Aug 2023
NormKD: Normalized Logits for Knowledge Distillation Zhihao Chi Tu Zheng Hengjia Li Zheng Yang Boxi Wu Binbin Lin D. Cai 32 13 0 01 Aug 2023
AdvDiff: Generating Unrestricted Adversarial Examples using Diffusion Models Xuelong Dai Kaisheng Liang Bin Xiao DiffM 23 24 0 24 Jul 2023
Augmented Box Replay: Overcoming Foreground Shift for Incremental Object Detection Liu Yuyang Cong Yang Dipam Goswami Liu Xialei Joost van de Weijer CLL 23 24 0 23 Jul 2023
Cluster-aware Semi-supervised Learning: Relational Knowledge Distillation Provably Learns Clustering Yijun Dong Kevin Miller Qiuyu Lei Rachel A. Ward 30 4 0 20 Jul 2023
Cumulative Spatial Knowledge Distillation for Vision Transformers Borui Zhao Renjie Song Jiajun Liang 34 14 0 17 Jul 2023
DOT: A Distillation-Oriented Trainer Borui Zhao Quan Cui Renjie Song Jiajun Liang 27 6 0 17 Jul 2023
DreamTeacher: Pretraining Image Backbones with Deep Generative Models Daiqing Li Huan Ling Amlan Kar David Acuna Seung Wook Kim Karsten Kreis Antonio Torralba Sanja Fidler VLM DiffM 22 27 0 14 Jul 2023
The Staged Knowledge Distillation in Video Classification: Harmonizing Student Progress by a Complementary Weakly Supervised Framework Chao Wang Zhenghang Tang 36 1 0 11 Jul 2023
Customizing Synthetic Data for Data-Free Student Learning Shiya Luo Defang Chen Can Wang 14 2 0 10 Jul 2023
Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data Qing Xu Min-man Wu Xiaoli Li K. Mao Zhenghua Chen 19 5 0 07 Jul 2023
Review helps learn better: Temporal Supervised Knowledge Distillation Dongwei Wang Zhi Han Yanmei Wang Xi’ai Chen Baichen Liu Yandong Tang 60 1 0 03 Jul 2023
Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler Shaohui Lin Wenxuan Huang Jiao Xie Baochang Zhang Yunhang Shen Zhou Yu Jungong Han David Doermann 25 2 0 01 Jul 2023
Miniaturized Graph Convolutional Networks with Topologically Consistent Pruning H. Sahbi 28 0 0 30 Jun 2023
NCL++: Nested Collaborative Learning for Long-Tailed Visual Recognition Zichang Tan Jun Yu Li Jinhao Du Jun Wan Zhen Lei Guodong Guo VLM 35 21 0 29 Jun 2023
A Dimensional Structure based Knowledge Distillation Method for Cross-Modal Learning Hui Xiong Hongwei Dong Jingyao Wang J. Yu Wen-jie Zhai Changwen Zheng Fanjiang Xu Gang Hua 24 1 0 28 Jun 2023
Deep Transfer Learning for Intelligent Vehicle Perception: a Survey Xinyi Liu Jinlong Li Jin Ma Huiming Sun Zhigang Xu Tianyu Zhang Hongkai Yu 61 23 0 26 Jun 2023
Cross Architecture Distillation for Face Recognition Weisong Zhao Xiangyu Zhu Zhixiang He Xiaoyu Zhang Zhen Lei CVBM 22 6 0 26 Jun 2023
Feature Adversarial Distillation for Point Cloud Classification Yuxing Lee Wei-Chieh Wu 3DPC 27 2 0 25 Jun 2023
Categories of Response-Based, Feature-Based, and Relation-Based Knowledge Distillation Chuanguang Yang Xinqiang Yu Zhulin An Yongjun Xu VLM OffRL 91 23 0 19 Jun 2023
BPKD: Boundary Privileged Knowledge Distillation For Semantic Segmentation Liyang Liu Zihan Wang M. Phan Bowen Zhang Jinchao Ge Yifan Liu 32 9 0 13 Jun 2023
Enhanced Multimodal Representation Learning with Cross-modal KD Mengxi Chen Linyu Xing Yu Wang Ya Zhang 34 11 0 13 Jun 2023
Adaptive Multi-Teacher Knowledge Distillation with Meta-Learning Hailin Zhang Defang Chen Can Wang 20 13 0 11 Jun 2023
Deep Classifier Mimicry without Data Access Steven Braun Martin Mundt Kristian Kersting DiffM 16 4 0 03 Jun 2023