Do Deep Nets Really Need to be Deep?

21 December 2013

Lei Jimmy Ba

Papers citing "Do Deep Nets Really Need to be Deep?"

50 / 337 papers shown

Title
Optimizing LLMs for Resource-Constrained Environments: A Survey of Model Compression Techniques Sanjay Surendranath Girija Shashank Kapoor Lakshit Arora Dipen Pradhan Aman Raj Ankit Shetgaonkar 57 0 0 05 May 2025
Transfer Learning with Pre-trained Conditional Generative Models Shin'ya Yamaguchi Sekitoshi Kanai Atsutoshi Kumagai Daiki Chijiwa H. Kashima VLM CLL BDL DiffM 148 5 0 21 Feb 2025
TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models Makoto Shing Kou Misaki Han Bao Sho Yokoi Takuya Akiba VLM 57 1 0 28 Jan 2025
Learning an Adaptive and View-Invariant Vision Transformer for Real-Time UAV Tracking You Wu Yongxin Li Mengyuan Liu Xucheng Wang Xiangyang Yang Hengzhou Ye Dan Zeng Qijun Zhao Shuiwang Li 146 0 0 28 Dec 2024
Swiss Army Knife: Synergizing Biases in Knowledge from Vision Foundation Models for Multi-Task Learning Yuxiang Lu Shengcao Cao Yu-xiong Wang 55 1 0 18 Oct 2024
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation Mike Ranzinger Jon Barker Greg Heinrich Pavlo Molchanov Bryan Catanzaro Andrew Tao 42 5 0 02 Oct 2024
Distilling System 2 into System 1 Ping Yu Jing Xu Jason Weston Ilia Kulikov OffRL LRM 52 62 0 08 Jul 2024
A Label is Worth a Thousand Images in Dataset Distillation Tian Qin Zhiwei Deng David Alvarez-Melis DD 94 10 0 15 Jun 2024
DistilDoc: Knowledge Distillation for Visually-Rich Document Applications Jordy Van Landeghem Subhajit Maity Ayan Banerjee Matthew Blaschko Marie-Francine Moens Josep Lladós Sanket Biswas 50 2 0 12 Jun 2024
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework Junxian Li Bin Shi Erfei Cui Hua Wei Qinghua Zheng 49 0 0 02 Mar 2024
GraphKD: Exploring Knowledge Distillation Towards Document Object Detection with Structured Graph Creation Ayan Banerjee Sanket Biswas Josep Lladós Umapada Pal 43 1 0 17 Feb 2024
Logit Poisoning Attack in Distillation-based Federated Learning and its Countermeasures Yonghao Yu Shunan Zhu Jinglu Hu AAML FedML 29 0 0 31 Jan 2024
Mutual Distillation Learning For Person Re-Identification Huiyuan Fu Kuilong Cui Chuanming Wang Mengshi Qi Huadong Ma 40 0 0 12 Jan 2024
The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction Pratyusha Sharma Jordan T. Ash Dipendra Kumar Misra LRM 19 78 0 21 Dec 2023
Mixed Distillation Helps Smaller Language Model Better Reasoning Chenglin Li Qianglong Chen Liangyue Li Wang Caiyu Yicheng Li Zhang Yin Yin Zhang LRM 41 12 0 17 Dec 2023
AM-RADIO: Agglomerative Vision Foundation Model -- Reduce All Domains Into One Michael Ranzinger Greg Heinrich Jan Kautz Pavlo Molchanov VLM 44 42 0 10 Dec 2023
DONUT-hole: DONUT Sparsification by Harnessing Knowledge and Optimizing Learning Efficiency Azhar Shaikh Michael Cochez Denis Diachkov Michiel de Rijcke Sahar Yousefi 25 0 0 09 Nov 2023
SCAAT: Improving Neural Network Interpretability via Saliency Constrained Adaptive Adversarial Training Rui Xu Wenkang Qin Peixiang Huang Hao Wang Lin Luo FAtt AAML 28 2 0 09 Nov 2023
ViPE: Visualise Pretty-much Everything Hassan Shahmohammadi Adhiraj Ghosh Hendrik P. A. Lensch DiffM 28 1 0 16 Oct 2023
Foreground Object Search by Distilling Composite Image Feature Bo Zhang Jiacheng Sui Li Niu 27 5 0 09 Aug 2023
Exploring the Lottery Ticket Hypothesis with Explainability Methods: Insights into Sparse Network Performance Shantanu Ghosh Kayhan Batmanghelich 30 0 0 07 Jul 2023
Self-Distilled Masked Auto-Encoders are Efficient Video Anomaly Detectors Nicolae-Cătălin Ristea Florinel-Alin Croitoru Radu Tudor Ionescu Marius Popescu Fahad Shahbaz Khan M. Shah ViT 26 20 0 21 Jun 2023
Interpretable Differencing of Machine Learning Models Swagatam Haldar Diptikalyan Saha Dennis L. Wei Rahul Nair Elizabeth M. Daly 16 1 0 10 Jun 2023
ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Models Suzanna Parkinson Greg Ongie Rebecca Willett 65 6 0 24 May 2023
Learning Interpretable Style Embeddings via Prompting LLMs Ajay Patel D. Rao Ansh Kothary Kathleen McKeown Chris Callison-Burch 37 24 0 22 May 2023
Analyzing Compression Techniques for Computer Vision Maniratnam Mandal Imran Khan 27 1 0 14 May 2023
Similarity of Neural Network Models: A Survey of Functional and Representational Measures Max Klabunde Tobias Schumacher M. Strohmaier Florian Lemmerich 58 64 0 10 May 2023
Target-Side Augmentation for Document-Level Machine Translation Guangsheng Bao Zhiyang Teng Yue Zhang 29 10 0 08 May 2023
Madvex: Instrumentation-based Adversarial Attacks on Machine Learning Malware Detection Yang Cai Felix Mächtle C. Daskalakis Volodymyr Bezsmertnyi T. Eisenbarth AAML 31 7 0 04 May 2023
A Survey on Solving and Discovering Differential Equations Using Deep Neural Networks Hyeonjung Jung Jung Jayant Gupta B. Jayaprakash Matthew J. Eagon Harish Selvam Carl Molnar W. Northrop Shashi Shekhar AI4CE 35 5 0 26 Apr 2023
Self-Distillation for Gaussian Process Regression and Classification Kenneth Borup L. Andersen 11 2 0 05 Apr 2023
DIME-FM: DIstilling Multimodal and Efficient Foundation Models Ximeng Sun Pengchuan Zhang Peizhao Zhang Hardik Shah Kate Saenko Xide Xia VLM 25 20 0 31 Mar 2023
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society Ge Li Hasan Hammoud Hani Itani Dmitrii Khizbullin Guohao Li SyDa ALM 35 412 0 31 Mar 2023
Exploiting Unlabelled Photos for Stronger Fine-Grained SBIR Aneeshan Sain A. Bhunia Subhadeep Koley Pinaki Nath Chowdhury Soumitri Chattopadhyay Tao Xiang Yi-Zhe Song 28 18 0 24 Mar 2023
MV-MR: multi-views and multi-representations for self-supervised learning and knowledge distillation Vitaliy Kinakh M. Drozdova Slava Voloshynovskiy 40 1 0 21 Mar 2023
Neural Architecture Search for Effective Teacher-Student Knowledge Transfer in Language Models Aashka Trivedi Takuma Udagawa Michele Merler Yikang Shen Yousef El-Kurdi Bishwaranjan Bhattacharjee 30 7 0 16 Mar 2023
Distilling Calibrated Student from an Uncalibrated Teacher Ishan Mishra Sethu Vamsi Krishna Deepak Mishra FedML 40 2 0 22 Feb 2023
Deep Learning Meets Sparse Regularization: A Signal Processing Perspective Rahul Parhi Robert D. Nowak 40 25 0 23 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 53 121 0 17 Jan 2023
In-Sensor & Neuromorphic Computing are all you need for Energy Efficient Computer Vision Gourav Datta Zeyu Liu Md. Abdullah-Al Kaiser Souvik Kundu Joe Mathai Zihan Yin Ajey P. Jacob Akhilesh R. Jaiswal P. Beerel 26 12 0 21 Dec 2022
Gait Recognition Using 3-D Human Body Shape Inference Haidong Zhu Zhao-Heng Zheng Ramkant Nevatia CVBM 3DH 25 23 0 18 Dec 2022
Co-training $2^L$ Submodels for Visual Recognition Hugo Touvron Matthieu Cord Maxime Oquab Piotr Bojanowski Jakob Verbeek Hervé Jégou VLM 35 9 0 09 Dec 2022
Investigating certain choices of CNN configurations for brain lesion segmentation Masoomeh Rahimpour A. Radwan Henri Vandermeulen S. Sunaert K. Goffin M. Koole 50 1 0 02 Dec 2022
Decentralized Learning with Multi-Headed Distillation A. Zhmoginov Mark Sandler Nolan Miller Gus Kristiansen Max Vladymyrov FedML 40 4 0 28 Nov 2022
Lightning Fast Video Anomaly Detection via Adversarial Knowledge Distillation Florinel-Alin Croitoru Nicolae-Cătălin Ristea D. Dascalescu Radu Tudor Ionescu Fahad Shahbaz Khan M. Shah 39 2 0 28 Nov 2022
Compressing Transformer-based self-supervised models for speech processing Tzu-Quan Lin Tsung-Huan Yang Chun-Yao Chang Kuang-Ming Chen Tzu-hsun Feng Hung-yi Lee Hao Tang 40 6 0 17 Nov 2022
Partial Binarization of Neural Networks for Budget-Aware Efficient Learning Udbhav Bamba Neeraj Anand Saksham Aggarwal Dilip K Prasad D. K. Gupta MQ 26 0 0 12 Nov 2022
Avoid Overthinking in Self-Supervised Models for Speech Recognition Dan Berrebbi Brian Yan Shinji Watanabe LRM 23 4 0 01 Nov 2022
Pixel-Wise Contrastive Distillation Junqiang Huang Zichao Guo 42 4 0 01 Nov 2022
QuaLA-MiniLM: a Quantized Length Adaptive MiniLM Shira Guskin Moshe Wasserblat Chang Wang Haihao Shen MQ 16 2 0 31 Oct 2022