Reducing Transformer Depth on Demand with Structured Dropout

25 September 2019

Angela Fan

Papers citing "Reducing Transformer Depth on Demand with Structured Dropout"

50 / 400 papers shown

Title
Pit One Against Many: Leveraging Attention-head Embeddings for Parameter-efficient Multi-head Attention Huiyin Xue Nikolaos Aletras 30 0 0 11 Oct 2023
Can pruning make Large Language Models more efficient? Sia Gholami Marwan Omar 28 12 0 06 Oct 2023
Federated Learning with Differential Privacy for End-to-End Speech Recognition Martin Pelikan Sheikh Shams Azam Vitaly Feldman Jan Honza Silovsky Kunal Talwar Tatiana Likhomanenko 42 8 0 29 Sep 2023
AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition Andrew Rouditchenko R. Collobert Tatiana Likhomanenko VLM 27 3 0 29 Sep 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 28 15 0 28 Sep 2023
CoMFLP: Correlation Measure based Fast Search on ASR Layer Pruning W. Liu Zhiyuan Peng Tan Lee 13 1 0 21 Sep 2023
Sorted LLaMA: Unlocking the Potential of Intermediate Layers of Large Language Models for Dynamic Inference Parsa Kavehzadeh Mojtaba Valipour Marzieh S. Tahaei Ali Ghodsi Boxing Chen Mehdi Rezagholizadeh 35 6 0 16 Sep 2023
Neurons in Large Language Models: Dead, N-gram, Positional Elena Voita Javier Ferrando Christoforos Nalmpantis MILM 32 45 0 09 Sep 2023
Enhancing Deep Learning Models through Tensorization: A Comprehensive Survey and Framework Manal Helal 19 0 0 05 Sep 2023
Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models Qiong Wu Wei Yu Yiyi Zhou Shubin Huang Xiaoshuai Sun Rongrong Ji VLM 26 7 0 04 Sep 2023
SortedNet: A Scalable and Generalized Framework for Training Modular Deep Neural Networks Mojtaba Valipour Mehdi Rezagholizadeh Hossein Rajabzadeh Parsa Kavehzadeh Marzieh S. Tahaei Boxing Chen Ali Ghodsi 15 1 0 01 Sep 2023
$$\rm SP^3$: Enhancing Structured Pruning via PCA Projection$ $\rm SP^3$ : Enhancing Structured Pruning via PCA Projection Yuxuan Hu Jing Zhang Zhe Zhao Chengliang Zhao Xiaodong Chen Cuiping Li Hong Chen 35 1 0 31 Aug 2023
Discrete Prompt Compression with Reinforcement Learning Hoyoun Jung Kyung-Joong Kim 29 24 0 17 Aug 2023
DPBERT: Efficient Inference for BERT based on Dynamic Planning Weixin Wu H. Zhuo 6 0 0 26 Jul 2023
Gradient Sparsification For Masked Fine-Tuning of Transformers J. Ó. Neill Sourav Dutta 21 0 0 19 Jul 2023
A Survey of Techniques for Optimizing Transformer Inference Krishna Teja Chitty-Venkata Sparsh Mittal M. Emani V. Vishwanath Arun Somani 43 62 0 16 Jul 2023
No Train No Gain: Revisiting Efficient Training Algorithms For Transformer-based Language Models Jean Kaddour Oscar Key Piotr Nawrot Pasquale Minervini Matt J. Kusner 20 41 0 12 Jul 2023
Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models James OÑeill Sourav Dutta VLM MQ 40 1 0 12 Jul 2023
Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decoding Seongjun Yang Gibbeum Lee Jaewoong Cho Dimitris Papailiopoulos Kangwook Lee 23 33 0 12 Jul 2023
Learning to Group Auxiliary Datasets for Molecule Ting Huang Ziniu Hu Rex Ying 11 0 0 08 Jul 2023
When Does Confidence-Based Cascade Deferral Suffice? Wittawat Jitkrittum Neha Gupta A. Menon Harikrishna Narasimhan A. S. Rawat Surinder Kumar 22 18 0 06 Jul 2023
Training Transformers with 4-bit Integers Haocheng Xi Changhao Li Jianfei Chen Jun Zhu MQ 25 47 0 21 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 62 355 0 20 Jun 2023
LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximation Yixiao Li Yifan Yu Qingru Zhang Chen Liang Pengcheng He Weizhu Chen Tuo Zhao 44 66 0 20 Jun 2023
MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition Yuchen Hu Chen Chen Ruizhe Li Heqing Zou Chng Eng Siong GAN 42 9 0 18 Jun 2023
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Cheng Chen Chengwei Qin Qiu-shi Zhu E. Chng 29 5 0 18 Jun 2023
SqueezeLLM: Dense-and-Sparse Quantization Sehoon Kim Coleman Hooper A. Gholami Zhen Dong Xiuyu Li Sheng Shen Michael W. Mahoney Kurt Keutzer MQ 29 167 0 13 Jun 2023
Revisiting Token Pruning for Object Detection and Instance Segmentation Yifei Liu Mathias Gehrig Nico Messikommer Marco Cannici Davide Scaramuzza ViT VLM 37 24 0 12 Jun 2023
Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency Yuxuan Wang Hong Lyu 24 2 0 05 Jun 2023
Modular Transformers: Compressing Transformers into Modularized Layers for Flexible Efficient Inference Wangchunshu Zhou Ronan Le Bras Yejin Choi 11 0 0 04 Jun 2023
The Information Pathways Hypothesis: Transformers are Dynamic Self-Ensembles Md Shamim Hussain Mohammed J. Zaki D. Subramanian 37 3 0 02 Jun 2023
DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models Yifan Peng Yui Sudo Muhammad Shakeel Shinji Watanabe 24 37 0 28 May 2023
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers Dachuan Shi Chaofan Tao Anyi Rao Zhendong Yang Chun Yuan Jiaqi Wang VLM 30 22 0 27 May 2023
Revisiting Token Dropping Strategy in Efficient BERT Pretraining Qihuang Zhong Liang Ding Juhua Liu Xuebo Liu Min Zhang Bo Du Dacheng Tao VLM 34 9 0 24 May 2023
Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model Fine-tuning Zhen-Ru Zhang Chuanqi Tan Haiyang Xu Chengyu Wang Jun Huang Songfang Huang 25 29 0 24 May 2023
SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models Zekun Wang Jingchang Chen Wangchunshu Zhou Haichao Zhu Jiafeng Liang Liping Shan Ming Liu Dongliang Xu Qing Yang Bing Qin VLM 24 4 0 24 May 2023
Just CHOP: Embarrassingly Simple LLM Compression A. Jha Tom Sherborne Evan Pete Walsh Dirk Groeneveld Emma Strubell Iz Beltagy 28 3 0 24 May 2023
PruMUX: Augmenting Data Multiplexing with Model Compression Yushan Su Vishvak Murahari Karthik Narasimhan Keqin Li 25 3 0 24 May 2023
Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator Ziwei He Meng-Da Yang Minwei Feng Jingcheng Yin Xinbing Wang Jingwen Leng Zhouhan Lin ViT 35 11 0 24 May 2023
One-stop Training of Multiple Capacity Models Lan Jiang Haoyang Huang Dongdong Zhang R. Jiang Furu Wei 28 0 0 23 May 2023
Infor-Coef: Information Bottleneck-based Dynamic Token Downsampling for Compact and Efficient language model Wenxin Tan 17 0 0 21 May 2023
F-PABEE: Flexible-patience-based Early Exiting for Single-label and Multi-label text Classification Tasks Xiangxiang Gao Wei-wei Zhu Jiasheng Gao Congrui Yin VLM 26 12 0 21 May 2023
Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with LLMs Pranjal Aggarwal Aman Madaan Yiming Yang Mausam LRM 28 36 0 19 May 2023
LLM-Pruner: On the Structural Pruning of Large Language Models Xinyin Ma Gongfan Fang Xinchao Wang 30 364 0 19 May 2023
Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation Yuxin Ren Zi-Qi Zhong Xingjian Shi Yi Zhu Chun Yuan Mu Li 24 7 0 16 May 2023
Parameter-Efficient Fine-Tuning with Layer Pruning on Free-Text Sequence-to-Sequence Modeling Y. Zhu Xuebing Yang Yuanyuan Wu Wensheng Zhang MedIm 20 2 0 15 May 2023
The EarlyBIRD Catches the Bug: On Exploiting Early Layers of Encoder Models for More Efficient Code Classification Anastasiia Grishina Max Hort Leon Moonen 22 6 0 08 May 2023
Transformer-based models and hardware acceleration analysis in autonomous driving: A survey J. Zhong Zheng Liu Xiangshan Chen ViT 44 17 0 21 Apr 2023
Eliciting Latent Predictions from Transformers with the Tuned Lens Nora Belrose Zach Furman Logan Smith Danny Halawi Igor V. Ostrovsky Lev McKinney Stella Biderman Jacob Steinhardt 22 193 0 14 Mar 2023
I3D: Transformer architectures with input-dependent dynamic depth for speech recognition Yifan Peng Jaesong Lee Shinji Watanabe 27 19 0 14 Mar 2023