Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer

12 December 2016

Papers citing "Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer"

50 / 1,157 papers shown

Title
Less or More From Teacher: Exploiting Trilateral Geometry For Knowledge Distillation Chengming Hu Haolun Wu Xuan Li Chen Ma Xi Chen Jun Yan Boyu Wang Xue Liu 35 3 0 22 Dec 2023
StableKD: Breaking Inter-block Optimization Entanglement for Stable Knowledge Distillation Shiu-hong Kao Jierun Chen S.-H. Gary Chan 27 0 0 20 Dec 2023
Decoupled Knowledge with Ensemble Learning for Online Distillation Baitan Shao Ying Chen 26 0 0 18 Dec 2023
Polyper: Boundary Sensitive Polyp Segmentation Hao Shao Yang Zhang Qibin Hou 21 10 0 14 Dec 2023
RdimKD: Generic Distillation Paradigm by Dimensionality Reduction Yi Guo Yiqian He Xiaoyang Li Haotong Qin Van Tung Pham Yang Zhang Shouda Liu 51 1 0 14 Dec 2023
Generative Model-based Feature Knowledge Distillation for Action Recognition Guiqin Wang Peng Zhao Yanjiang Shi Cong Zhao Shusen Yang VLM 52 3 0 14 Dec 2023
MCANet: Medical Image Segmentation with Multi-Scale Cross-Axis Attention Hao Shao Quansheng Zeng Qibin Hou Jufeng Yang 51 13 0 14 Dec 2023
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One Michael Ranzinger Greg Heinrich Jan Kautz Pavlo Molchanov VLM 49 42 0 10 Dec 2023
Augmentation-Free Dense Contrastive Knowledge Distillation for Efficient Semantic Segmentation Jiawei Fan Chao Li Xiaolong Liu Meina Song Anbang Yao 36 5 0 07 Dec 2023
On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigm Peng Sun Bei Shi Daiwei Yu Tao Lin DD 29 40 0 06 Dec 2023
The Efficiency Spectrum of Large Language Models: An Algorithmic Survey Tianyu Ding Tianyi Chen Haidong Zhu Jiachen Jiang Yiqi Zhong Jinxin Zhou Guangzhi Wang Zhihui Zhu Ilya Zharkov Luming Liang 31 22 0 01 Dec 2023
Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task-specific Models Raviteja Vemulapalli Hadi Pouransari Fartash Faghri Sachin Mehta Mehrdad Farajtabar Mohammad Rastegari Oncel Tuzel 43 7 0 30 Nov 2023
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS Zhiwen Fan Kevin Wang Kairun Wen Zehao Zhu Dejia Xu Zhangyang Wang 3DGS 39 187 0 28 Nov 2023
Rethinking Intermediate Layers design in Knowledge Distillation for Kidney and Liver Tumor Segmentation Vandan Gorade Sparsh Mittal Debesh Jha Ulas Bagci 25 3 0 28 Nov 2023
Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization Jixuan Leng Yijiang Li Haohan Wang VLM 37 0 0 26 Nov 2023
BinaryHPE: 3D Human Pose and Shape Estimation via Binarization Zhiteng Li Yulun Zhang Jing Lin Haotong Qin Jinjin Gu Xin Yuan Linghe Kong Xiaokang Yang 3DH 42 1 0 24 Nov 2023
Maximizing Discrimination Capability of Knowledge Distillation with Energy Function Seonghak Kim Gyeongdo Ham Suin Lee Donggon Jang Daeshik Kim 34 4 0 24 Nov 2023
Education distillation:getting student models to learn in shcools Ling Feng Danyang Li Tianhao Wu Xuliang Duan Xuliang Duan FedML 30 0 0 23 Nov 2023
Robustness-Reinforced Knowledge Distillation with Correlation Distance and Network Pruning Seonghak Kim Gyeongdo Ham Yucheol Cho Daeshik Kim 30 3 0 23 Nov 2023
Text Representation Distillation via Information Bottleneck Principle Yanzhao Zhang Dingkun Long Zehan Li Pengjun Xie 19 2 0 09 Nov 2023
Contactless Fingerprint Biometric Anti-Spoofing: An Unsupervised Deep Learning Approach Banafsheh Adami Nima Karimian AAML 23 5 0 07 Nov 2023
Cross-Level Distillation and Feature Denoising for Cross-Domain Few-Shot Classification Hao Zheng Runqi Wang Jianzhuang Liu Asako Kanezaki 44 7 0 04 Nov 2023
Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models Andy Zhou Jindong Wang Yu-xiong Wang Haohan Wang VLM 57 6 0 02 Nov 2023
One-for-All: Bridge the Gap Between Heterogeneous Architectures in Knowledge Distillation Zhiwei Hao Jianyuan Guo Kai Han Yehui Tang Han Hu Yunhe Wang Chang Xu 48 59 0 30 Oct 2023
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model Karsten Roth Lukas Thede Almut Sophia Koepke Oriol Vinyals Olivier J. Hénaff Zeynep Akata AAML 30 12 0 26 Oct 2023
Understanding the Effects of Projectors in Knowledge Distillation Yudong Chen Sen Wang Jiajun Liu Xuwei Xu Frank de Hoog Brano Kusy Zi Huang 26 0 0 26 Oct 2023
ABKD: Graph Neural Network Compression with Attention-Based Knowledge Distillation Anshul Ahluwalia Rohit Das Payman Behnam Alind Khare Pan Li Alexey Tumanov 35 2 0 24 Oct 2023
How a student becomes a teacher: learning and forgetting through Spectral methods Lorenzo Giambagli L. Buffoni Lorenzo Chicchi Duccio Fanelli 19 7 0 19 Oct 2023
Exploiting User Comments for Early Detection of Fake News Prior to Users' Commenting Qiong Nan Qiang Sheng Juan Cao Yongchun Zhu Danding Wang Guang Yang Jintao Li Kai Shu 52 8 0 16 Oct 2023
SeUNet-Trans: A Simple yet Effective UNet-Transformer Model for Medical Image Segmentation Tan-Hanh Pham Xianqi Li Kim-Doang Nguyen MedIm ViT 26 9 0 16 Oct 2023
MAC: ModAlity Calibration for Object Detection Yutian Lei Jun Liu Dong Huang ObjD 20 0 0 14 Oct 2023
Towards the Fundamental Limits of Knowledge Transfer over Finite Domains Qingyue Zhao Banghua Zhu 41 4 0 11 Oct 2023
Bidirectional Knowledge Reconfiguration for Lightweight Point Cloud Analysis Peipei Li Xing Cui Yibo Hu Man Zhang Ting Yao Tao Mei 33 0 0 08 Oct 2023
LumiNet: The Bright Side of Perceptual Knowledge Distillation Md. Ismail Hossain M. M. L. Elahi Sameera Ramasinghe A. Cheraghian Fuad Rahman Nabeel Mohammed Shafin Rahman 39 1 0 05 Oct 2023
Improving Knowledge Distillation with Teacher's Explanation S. Chowdhury Ben Liang A. Tizghadam Ilijc Albanese FAtt 19 0 0 04 Oct 2023
Can a student Large Language Model perform as well as it's teacher? Sia Gholami Marwan Omar 29 11 0 03 Oct 2023
Distilling Influences to Mitigate Prediction Churn in Graph Neural Networks Andreas Roth Thomas Liebig 52 0 0 02 Oct 2023
DataDAM: Efficient Dataset Distillation with Attention Matching A. Sajedi Samir Khaki Ehsan Amjadian Lucy Z. Liu Y. Lawryshyn Konstantinos N. Plataniotis DD 53 60 0 29 Sep 2023
Unsupervised Pretraining for Fact Verification by Language Model Distillation A. Bazaga Pietro Lio Bo Dai HILM 38 2 0 28 Sep 2023
An Enhanced Low-Resolution Image Recognition Method for Traffic Environments Flavio Figueiredo Zhenhai Gao 11 0 0 28 Sep 2023
Noise-Tolerant Unsupervised Adapter for Vision-Language Models Eman Ali Dayan Guan Muhammad Haris Khan Abdulmotaleb Elsaddik VLM 24 0 0 26 Sep 2023
LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference Hongwu Peng Ran Ran Yukui Luo Jiahui Zhao Shaoyi Huang ... Tong Geng Chenghong Wang Xiaolin Xu Wujie Wen Caiwen Ding 34 36 0 25 Sep 2023
Data Upcycling Knowledge Distillation for Image Super-Resolution Yun-feng Zhang Wei Li Simiao Li Hanting Chen Zhaopeng Tu Wenjun Wang Bingyi Jing Hai-lin Wang Jie Hu 35 3 0 25 Sep 2023
Targeted Activation Penalties Help CNNs Ignore Spurious Signals Dekai Zhang Matthew Williams Francesca Toni AAML 20 1 0 22 Sep 2023
Weight Averaging Improves Knowledge Distillation under Domain Shift Valeriy Berezovskiy Nikita Morozov MoMe 33 1 0 20 Sep 2023
Heterogeneous Generative Knowledge Distillation with Masked Image Modeling Ziming Wang Shumin Han Xiaodi Wang Jing Hao Xianbin Cao Baochang Zhang VLM 32 0 0 18 Sep 2023
One-stage Modality Distillation for Incomplete Multimodal Learning Shicai Wei Yang Luo Chunbo Luo 23 0 0 15 Sep 2023
Two-Step Knowledge Distillation for Tiny Speech Enhancement Rayan Daod Nathoo M. Kegler Marko Stamenovic 19 4 0 15 Sep 2023
Towards Comparable Knowledge Distillation in Semantic Image Segmentation Onno Niemann Christopher Vox Thorben Werner VLM 25 1 0 07 Sep 2023
Knowledge Distillation Layer that Lets the Student Decide Ada Gorgun Y. Z. Gürbüz A. Aydin Alatan 31 0 0 06 Sep 2023