Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search

14 October 2020

Papers citing "Length-Adaptive Transformer: Train Once with Length Drop, Use Anytime with Search"

27 / 27 papers shown

Title
Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps Jie Ou Jinyu Guo Shuaihong Jiang Zhaokun Wang Libo Qin Shunyu Yao Wenhong Tian 3DV 4 0 0 19 May 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 36 0 0 06 Apr 2025
Beyond Matryoshka: Revisiting Sparse Coding for Adaptive Representation Tiansheng Wen Yifei Wang Zequn Zeng Zhong Peng Yudi Su Xinyang Liu Bo Chen Hongwei Liu Stefanie Jegelka Chenyu You CLL 71 3 0 03 Mar 2025
CipherPrune: Efficient and Scalable Private Transformer Inference Yancheng Zhang Jinbao Xue Mengxin Zheng Mimi Xie Mingzhe Zhang Lei Jiang Qian Lou 64 2 0 24 Feb 2025
Training Noise Token Pruning Mingxing Rao Bohan Jiang Daniel Moyer ViT 74 0 0 27 Nov 2024
ElasticTok: Adaptive Tokenization for Image and Video Wilson Yan Matei A. Zaharia Volodymyr Mnih Pieter Abbeel Aleksandra Faust Hao Liu VGen 46 6 0 10 Oct 2024
LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models Huiqiang Jiang Qianhui Wu Chin-Yew Lin Yuqing Yang Lili Qiu 34 101 0 09 Oct 2023
Zero-TPrune: Zero-Shot Token Pruning through Leveraging of the Attention Graph in Pre-Trained Transformers Hongjie Wang Bhishma Dedhia N. Jha ViT VLM 44 26 0 27 May 2023
Do We Really Need a Large Number of Visual Prompts? Youngeun Kim Yuhang Li Abhishek Moitra Ruokai Yin Priyadarshini Panda VLM VPVLM 43 5 0 26 May 2023
Lifting the Curse of Capacity Gap in Distilling Language Models Chen Zhang Yang Yang Jiahao Liu Jingang Wang Yunsen Xian Benyou Wang Dawei Song MoE 32 19 0 20 May 2023
Full Stack Optimization of Transformer Inference: a Survey Sehoon Kim Coleman Hooper Thanakul Wattanawong Minwoo Kang Ruohan Yan ... Qijing Huang Kurt Keutzer Michael W. Mahoney Y. Shao A. Gholami MQ 36 101 0 27 Feb 2023
DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing Conglong Li Z. Yao Xiaoxia Wu Minjia Zhang Connor Holmes Cheng Li Yuxiong He 27 24 0 07 Dec 2022
Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans? Neeraj Varshney Man Luo Chitta Baral RALM 21 11 0 23 Nov 2022
Random-LTD: Random and Layerwise Token Dropping Brings Efficient Training for Large-scale Transformers Z. Yao Xiaoxia Wu Conglong Li Connor Holmes Minjia Zhang Cheng-rong Li Yuxiong He 28 11 0 17 Nov 2022
QuaLA-MiniLM: a Quantized Length Adaptive MiniLM Shira Guskin Moshe Wasserblat Chang Wang Haihao Shen MQ 13 2 0 31 Oct 2022
Token Merging: Your ViT But Faster Daniel Bolya Cheng-Yang Fu Xiaoliang Dai Peizhao Zhang Christoph Feichtenhofer Judy Hoffman MoMe 51 417 0 17 Oct 2022
SpanDrop: Simple and Effective Counterfactual Learning for Long Sequences Peng Qi Guangtao Wang Jing Huang 24 0 0 03 Aug 2022
Building an Efficiency Pipeline: Commutativity and Cumulativeness of Efficiency Operators for Transformers Ji Xin Raphael Tang Zhiying Jiang Yaoliang Yu Jimmy J. Lin 18 1 0 31 Jul 2022
Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selection Xin Huang A. Khetan Rene Bidart Zohar Karnin 19 14 0 27 Mar 2022
Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention Zuzana Jelčicová Marian Verhelst 28 5 0 20 Mar 2022
A Study on Token Pruning for ColBERT Carlos Lassance Maroua Maachou Joohee Park S. Clinchant 30 13 0 13 Dec 2021
Learned Token Pruning for Transformers Sehoon Kim Sheng Shen D. Thorsley A. Gholami Woosuk Kwon Joseph Hassoun Kurt Keutzer 17 145 0 02 Jul 2021
Shortformer: Better Language Modeling using Shorter Inputs Ofir Press Noah A. Smith M. Lewis 230 89 0 31 Dec 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,017 0 28 Jul 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 264 4,489 0 23 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,826 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018