Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 400 papers shown

Title
Position: Enough of Scaling LLMs! Lets Focus on Downscaling Ayan Sengupta Yash Goel Tanmoy Chakraborty 34 0 0 02 May 2025
Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning Yuanbing Ouyang Yizhuo Liang Qingpeng Li Xinfei Guo Yiming Luo Di Wu Hao Wang Yushan Pan ViT VLM 73 0 0 25 Apr 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 34 0 0 06 Apr 2025
Efficient Token Compression for Vision Transformer with Spatial Information Preserved Junzhu Mao Yang Shen Jinyang Guo Yazhou Yao Xiansheng Hua ViT 33 0 0 30 Mar 2025
Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices Z. Wang Yexiao He Zheyu Shen Yu Li Guoheng Sun Myungjin Lee Ang Li 48 0 0 19 Mar 2025
Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruning Lizhen Xu Xiuxiu Bai Xiaojun Jia Jianwu Fang Shanmin Pang 61 0 0 13 Mar 2025
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation Tiansheng Wen Yifei Wang Zequn Zeng Zhong Peng Yudi Su Xinyang Liu Bo Chen Hongwei Liu Stefanie Jegelka Chenyu You CLL 66 2 0 03 Mar 2025
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks Nikolay Blagoev Lydia Yiyu Chen Oğuzhan Ersoy 55 0 0 27 Feb 2025
On Pruning State-Space LLMs Tamer Ghattas Michael Hassid Roy Schwartz 54 1 0 26 Feb 2025
Disentangling Visual Transformers: Patch-level Interpretability for Image Classification Guillaume Jeanneret Loïc Simon F. Jurie ViT 44 0 0 24 Feb 2025
Reasoning Bias of Next Token Prediction Training Pengxiao Lin Zhongwang Zhang Zhi-Qin John Xu LRM 80 1 0 21 Feb 2025
ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources Jason Wu Kang Yang Lance M. Kaplan Mani B. Srivastava 31 0 0 11 Feb 2025
LUNAR: LLM Unlearning via Neural Activation Redirection William F. Shen Xinchi Qiu Meghdad Kurmanji Alex Iacob Lorenzo Sani Yihong Chen Nicola Cancedda Nicholas D. Lane MU 53 1 0 11 Feb 2025
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Andrew Rouditchenko Saurabhchand Bhati Samuel Thomas Hilde Kuehne Rogerio Feris 106 1 0 03 Feb 2025
BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts Divya J. Bajpai M. Hanawal 65 0 0 02 Feb 2025
Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable Lizhen Xu Shanmin Pang Wenzhao Qiu Zehao Wu Xiuxiu Bai K. Mei Jianru Xue 69 1 0 03 Dec 2024
Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation Junchen Fu Xuri Ge Xin Xin Alexandros Karatzoglou Ioannis Arapakis Kaiwen Zheng Yongxin Ni J. Jose 23 0 0 05 Nov 2024
CopRA: A Progressive LoRA Training Strategy Zhan Zhuang Xiequn Wang Yulong Zhang Wei Li Yu Zhang Ying Wei 33 1 0 30 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 76 5 0 28 Oct 2024
DQRM: Deep Quantized Recommendation Models Yang Zhou Zhen Dong Ellick Chan Dhiraj Kalamkar Diana Marculescu Kurt Keutzer MQ 50 1 0 26 Oct 2024
Dynamic layer selection in decoder-only transformers Theodore Glavas Joud Chataoui Florence Regol Wassim Jabbour Antonios Valkanas Boris N. Oreshkin Mark J. Coates AI4CE 24 0 0 26 Oct 2024
Large Language Models Are Overparameterized Text Encoders Thennal D K Tim Fischer Chris Biemann 38 2 0 18 Oct 2024
FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction Akriti Jain Saransh Sharma Koyel Mukherjee Soumyabrata Pal 26 1 0 16 Oct 2024
Persistent Topological Features in Large Language Models Yuri Gardinazzi Giada Panerai Karthik Viswanathan A. Ansuini Alberto Cazzaniga Matteo Biagetti 39 2 0 14 Oct 2024
Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models Fei Wang Li Shen Liang Ding Chao Xue Ye Liu Changxing Ding 30 0 0 13 Oct 2024
Token Pruning using a Lightweight Background Aware Vision Transformer Sudhakar Sah Ravish Kumar Honnesh Rohmetra Ehsan Saboori ViT 21 0 0 12 Oct 2024
ElasticTok: Adaptive Tokenization for Image and Video Wilson Yan Matei A. Zaharia Volodymyr Mnih Pieter Abbeel Aleksandra Faust Hao Liu VGen 43 6 0 10 Oct 2024
Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See Phu Pham Phu Pham Kun Wan Yu-Jhe Li Zeliang Zhang Daniel Miranda Ajinkya Kale Ajinkya Kale Chenliang Xu 27 5 0 08 Oct 2024
Towards Building Efficient Sentence BERT Models using Layer Pruning Anushka Shelke Riya Savant Raviraj Joshi 27 0 0 21 Sep 2024
PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening Ruocheng Wu ZiEn Zhang ShangQi Deng YuLe Duan LiangJian Deng 45 0 0 11 Sep 2024
STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning Jaeseong Lee seung-won hwang Aurick Qiao Daniel F Campos Z. Yao Yuxiong He 25 2 0 10 Sep 2024
Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase Yicong Li Xing Guo Haohua Du 30 0 0 16 Aug 2024
Convexity-based Pruning of Speech Representation Models Teresa Dorszewski Lenka Tětková Lars Kai Hansen 25 2 0 16 Aug 2024
Cross-layer Attention Sharing for Large Language Models Yongyu Mu Yuzhang Wu Yuchun Fan Chenglong Wang Hengyu Li Qiaozhi He Murun Yang Tong Xiao Jingbo Zhu 40 5 0 04 Aug 2024
Accelerating Large Language Model Inference with Self-Supervised Early Exits Florian Valade LRM 36 1 0 30 Jul 2024
Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models Georgy Tyukin G. Dovonon Jean Kaddour Pasquale Minervini LRM 31 0 0 22 Jul 2024
Reconstruct the Pruned Model without Any Retraining Pingjie Wang Ziqing Fan Shengchao Hu Zhe Chen Yanfeng Wang Yu Wang 38 1 0 18 Jul 2024
Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition Jingjing Xu Wei Zhou Zijian Yang Eugen Beck Ralf Schlueter 34 1 0 10 Jul 2024
LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order Matthias Anton Freiberger Peter Kun A. Løvlie Sebastian Risi 36 0 0 05 Jul 2024
AMD: Automatic Multi-step Distillation of Large-scale Vision Models Cheng Han Qifan Wang S. Dianat Majid Rabbani Raghuveer M. Rao Yi Fang Qiang Guan Lifu Huang Dongfang Liu VLM 33 4 0 05 Jul 2024
The Remarkable Robustness of LLMs: Stages of Inference? Vedang Lad Wes Gurnee Max Tegmark 33 33 0 27 Jun 2024
BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks A. Ramesh Vignesh Ganapathiraman I. Laradji Mark W. Schmidt 31 1 0 25 Jun 2024
An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers Ashim Gupta Sina Mahdipour Saravani P. Sadayappan Vivek Srikumar 30 2 0 17 Jun 2024
Enhancing Domain Adaptation through Prompt Gradient Alignment Hoang Phan Lam C. Tran Quyen Tran Trung Le 52 0 0 13 Jun 2024
DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models T. Lin Hung-yi Lee Hao Tang 40 1 0 08 Jun 2024
VTrans: Accelerating Transformer Compression with Variational Information Bottleneck based Pruning Oshin Dutta Ritvik Gupta Sumeet Agarwal 39 1 0 07 Jun 2024
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning Naibin Gu Peng Fu Xiyu Liu Bowen Shen Zheng-Shen Lin Weiping Wang 30 6 0 06 Jun 2024
Learning-to-Cache: Accelerating Diffusion Transformer via Layer Caching Xinyin Ma Gongfan Fang Michael Bi Mi Xinchao Wang 55 30 0 03 Jun 2024
SUBLLM: A Novel Efficient Architecture with Token Sequence Subsampling for LLM Quandong Wang Yuxuan Yuan Xiaoyu Yang Ruike Zhang Kang Zhao Wei Liu Jian Luan Daniel Povey Bin Wang 41 0 0 03 Jun 2024
S3D: A Simple and Cost-Effective Self-Speculative Decoding Scheme for Low-Memory GPUs Wei Zhong Manasa Bharadwaj 37 5 0 30 May 2024