Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 406 papers shown

Title
Accelerate 3D Object Detection Models via Zero-Shot Attention Key Pruning Lizhen Xu Xiuxiu Bai Xiaojun Jia Jianwu Fang Shanmin Pang 127 0 0 01 Jul 2025
Curriculum-Guided Layer Scaling for Language Model Pretraining Karanpartap Singh Neil Band Ehsan Adeli ALM LRM 37 0 0 13 Jun 2025
Adapt Once, Thrive with Updates: Transferable Parameter-Efficient Fine-Tuning on Evolving Base Models Naibin Gu Peng Fu Xiyu Liu Ke Ma Zheng Lin Weiping Wang 24 0 0 07 Jun 2025
Come Together, But Not Right Now: A Progressive Strategy to Boost Low-Rank Adaptation Zhan Zhuang Xiequn Wang Wei Li Yulong Zhang Qiushi Huang ... Yanbin Wei Yuhe Nie Kede Ma Yu Zhang Ying Wei 55 0 0 06 Jun 2025
CLaSp: In-Context Layer Skip for Self-Speculative Decoding Longze Chen Renke Shan Huiming Wang Lu Wang Ziqiang Liu Run Luo Jiawei Wang Hamid Alinejad-Rokny Min Yang 37 0 0 30 May 2025
Position: The Future of Bayesian Prediction Is Prior-Fitted Samuel G. Müller Arik Reuter Noah Hollmann David Rügamer Frank Hutter 30 0 0 29 May 2025
Is Attention Required for Transformer Inference? Explore Function-preserving Attention Replacement Yuxin Ren Maxwell D Collins Miao Hu Huanrui Yang 53 0 0 24 May 2025
AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformer Jiquan Shan Junxiao Wang Lifeng Zhao Liang Cai Hongyuan Zhang Ioannis Liritzis ViT 245 0 0 22 May 2025
Position: Enough of Scaling LLMs! Lets Focus on Downscaling Ayan Sengupta Ayan Sengupta Tanmoy Chakraborty 110 0 0 02 May 2025
Back to Fundamentals: Low-Level Visual Features Guided Progressive Token Pruning Yuanbing Ouyang Yizhuo Liang Qingpeng Li Xinfei Guo Yiming Luo Di Wu Hao Wang Yushan Pan ViT VLM 98 0 0 25 Apr 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 131 2 0 06 Apr 2025
Efficient Token Compression for Vision Transformer with Spatial Information Preserved Junzhu Mao Yang Shen Jinyang Guo Yazhou Yao Xiansheng Hua ViT 141 0 0 30 Mar 2025
Prada: Black-Box LLM Adaptation with Private Data on Resource-Constrained Devices Ziyi Wang Yexiao He Zheyu Shen Yu Li Guoheng Sun Myungjin Lee Ang Li 89 0 0 19 Mar 2025
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation Tiansheng Wen Yifei Wang Zequn Zeng Zhong Peng Yudi Su Xinyang Liu Bo Chen Hongwei Liu Stefanie Jegelka Chenyu You CLL 231 4 0 03 Mar 2025
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks Nikolay Blagoev Lydia Yiyu Chen Oğuzhan Ersoy 99 2 0 27 Feb 2025
On Pruning State-Space LLMs Tamer Ghattas Michael Hassid Roy Schwartz 94 2 0 26 Feb 2025
Disentangling Visual Transformers: Patch-level Interpretability for Image Classification Guillaume Jeanneret Loïc Simon F. Jurie ViT 158 0 0 24 Feb 2025
Reasoning Bias of Next Token Prediction Training Pengxiao Lin Zhongwang Zhang Zhi-Qin John Xu LRM 193 2 0 21 Feb 2025
LUNAR: LLM Unlearning via Neural Activation Redirection William F. Shen Xinchi Qiu Meghdad Kurmanji Alex Iacob Lorenzo Sani Yihong Chen Nicola Cancedda Nicholas D. Lane MU 124 6 0 11 Feb 2025
ADMN: A Layer-Wise Adaptive Multimodal Network for Dynamic Input Noise and Compute Resources Jason Wu Kang Yang Lance M. Kaplan Mani B. Srivastava 69 0 0 11 Feb 2025
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Andrew Rouditchenko Saurabhchand Bhati Samuel Thomas Hilde Kuehne Rogerio Feris 230 1 0 03 Feb 2025
BEEM: Boosting Performance of Early Exit DNNs using Multi-Exit Classifiers as Experts Divya J. Bajpai M. Hanawal 154 1 0 02 Feb 2025
Redundant Queries in DETR-Based 3D Detection Methods: Unnecessary and Prunable Lizhen Xu Shanmin Pang Wenzhao Qiu Zehao Wu Xiuxiu Bai K. Mei Jianru Xue 161 2 0 03 Dec 2024
Efficient and Effective Adaptation of Multimodal Foundation Models in Sequential Recommendation Junchen Fu Xuri Ge Xin Xin Alexandros Karatzoglou Ioannis Arapakis Kaiwen Zheng Yongxin Ni J. Jose 62 3 0 05 Nov 2024
CopRA: A Progressive LoRA Training Strategy Zhan Zhuang Xiequn Wang Yulong Zhang Wei Li Yu Zhang Ying Wei 114 1 0 30 Oct 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 137 7 0 28 Oct 2024
DQRM: Deep Quantized Recommendation Models Yang Zhou Zhen Dong Ellick Chan Dhiraj Kalamkar Diana Marculescu Kurt Keutzer MQ 102 1 0 26 Oct 2024
Dynamic layer selection in decoder-only transformers Theodore Glavas Joud Chataoui Florence Regol Wassim Jabbour Antonios Valkanas Boris N. Oreshkin Mark Coates AI4CE 83 1 0 26 Oct 2024
Large Language Models Are Overparameterized Text Encoders Thennal D K Tim Fischer Chris Biemann 85 2 0 18 Oct 2024
FiRST: Finetuning Router-Selective Transformers for Input-Adaptive Latency Reduction Akriti Jain Saransh Sharma Koyel Mukherjee Soumyabrata Pal 53 1 0 16 Oct 2024
Persistent Topological Features in Large Language Models Yuri Gardinazzi Giada Panerai Karthik Viswanathan A. Ansuini Alberto Cazzaniga Matteo Biagetti 156 2 0 14 Oct 2024
Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models Fei Wang Li Shen Liang Ding Chao Xue Ye Liu Changxing Ding 107 2 0 13 Oct 2024
Token Pruning using a Lightweight Background Aware Vision Transformer Sudhakar Sah Ravish Kumar Honnesh Rohmetra Ehsan Saboori ViT 122 1 0 12 Oct 2024
ElasticTok: Adaptive Tokenization for Image and Video Wilson Yan Matei A. Zaharia Volodymyr Mnih Pieter Abbeel Aleksandra Faust Hao Liu VGen 111 11 0 10 Oct 2024
Treat Visual Tokens as Text? But Your MLLM Only Needs Fewer Efforts to See Phu Pham Phu Pham Kun Wan Yu-Jhe Li Zeliang Zhang Daniel Miranda Ajinkya Kale Ajinkya Kale Chenliang Xu 96 9 0 08 Oct 2024
Towards Building Efficient Sentence BERT Models using Layer Pruning Anushka Shelke Riya Savant Raviraj Joshi 49 0 0 21 Sep 2024
PanAdapter: Two-Stage Fine-Tuning with Spatial-Spectral Priors Injecting for Pansharpening Ruocheng Wu ZiEn Zhang ShangQi Deng YuLe Duan LiangJian Deng 81 0 0 11 Sep 2024
STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning Jaeseong Lee seung-won hwang Aurick Qiao Daniel F Campos Z. Yao Yuxiong He 65 3 0 10 Sep 2024
Research on Personalized Compression Algorithm for Pre-trained Models Based on Homomorphic Entropy Increase Yicong Li Xing Guo Haohua Du 82 0 0 16 Aug 2024
Convexity-based Pruning of Speech Representation Models Teresa Dorszewski Lenka Tětková Lars Kai Hansen 95 2 0 16 Aug 2024
Cross-layer Attention Sharing for Large Language Models Yongyu Mu Yuzhang Wu Yuchun Fan Chenglong Wang Hengyu Li Qiaozhi He Murun Yang Tong Xiao Jingbo Zhu 85 5 0 04 Aug 2024
Accelerating Large Language Model Inference with Self-Supervised Early Exits Florian Valade LRM 72 1 0 30 Jul 2024
Attention Is All You Need But You Don't Need All Of It For Inference of Large Language Models Georgy Tyukin G. Dovonon Jean Kaddour Pasquale Minervini LRM 70 2 0 22 Jul 2024
Reconstruct the Pruned Model without Any Retraining Pingjie Wang Ziqing Fan Shengchao Hu Zhe Chen Yanfeng Wang Yu Wang 84 2 0 18 Jul 2024
Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognition Jingjing Xu Wei Zhou Zijian Yang Eugen Beck Ralf Schlueter 91 3 0 10 Jul 2024
LayerShuffle: Enhancing Robustness in Vision Transformers by Randomizing Layer Execution Order Matthias Anton Freiberger Peter Kun A. Løvlie Sebastian Risi 84 0 0 05 Jul 2024
AMD: Automatic Multi-step Distillation of Large-scale Vision Models Cheng Han Qifan Wang S. Dianat Majid Rabbani Raghuveer M. Rao Yi Fang Qiang Guan Lifu Huang Dongfang Liu VLM 82 5 0 05 Jul 2024
The Remarkable Robustness of LLMs: Stages of Inference? Vedang Lad Wes Gurnee Max Tegmark Max Tegmark 119 53 0 27 Jun 2024
BlockLLM: Memory-Efficient Adaptation of LLMs by Selecting and Optimizing the Right Coordinate Blocks A. Ramesh Vignesh Ganapathiraman I. Laradji Mark Schmidt 126 3 0 25 Jun 2024
An Empirical Investigation of Matrix Factorization Methods for Pre-trained Transformers Ashim Gupta Sina Mahdipour Saravani P. Sadayappan Vivek Srikumar 57 2 0 17 Jun 2024