Efficiently Scaling Transformer Inference

9 November 2022

Papers citing "Efficiently Scaling Transformer Inference"

50 / 61 papers shown

Title
Generative Pre-trained Autoregressive Diffusion Transformer Yuan Zhang Jiacheng Jiang Guoqing Ma Zhiying Lu Haoyang Huang Jianlong Yuan Nan Duan VGen 48 1 0 12 May 2025
Sparse Attention Remapping with Clustering for Efficient LLM Decoding on PIM Zehao Fan Garrett Gagnon Zhenyu Liu Liu Liu 29 0 0 09 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Yushen Chen Jiawei Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 38 0 0 05 May 2025
EntroLLM: Entropy Encoded Weight Compression for Efficient Large Language Model Inference on Edge Devices Arnab Sanyal Prithwish Mukherjee Gourav Datta Sandeep P. Chinchali MQ 186 0 0 05 May 2025
Cobra: Efficient Line Art COlorization with BRoAder References Junhao Zhuang Lingen Li Xuan Ju Zhaoyang Zhang Chun Yuan Ying Shan DiffM 67 0 0 16 Apr 2025
MSCCL++: Rethinking GPU Communication Abstractions for Cutting-edge AI Applications Aashaka Shah Abhinav Jangda Yangqiu Song Caio Rocha Changho Hwang ... Peng Cheng Qinghua Zhou Roshan Dathathri Saeed Maleki Ziyue Yang GNN 54 0 0 11 Apr 2025
Memory-efficient Streaming VideoLLMs for Real-time Procedural Video Understanding Dibyadip Chatterjee Edoardo Remelli Yale Song Bugra Tekin Abhay Mittal ... Shreyas Hampali Eric Sauser Shugao Ma Angela Yao Fadime Sener VLM 51 0 0 10 Apr 2025
Adaptive Layer-skipping in Pre-trained LLMs Xuan Luo Weizhi Wang Xifeng Yan 200 0 0 31 Mar 2025
Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song H. Shen 73 1 0 11 Mar 2025
Seesaw: High-throughput LLM Inference via Model Re-sharding Qidong Su Wei Zhao Xuelong Li Muralidhar Andoorveedu Chenhao Jiang Zhanda Zhu Kevin Song Christina Giannoula Gennady Pekhimenko LRM 77 0 0 09 Mar 2025
MALT Diffusion: Memory-Augmented Latent Transformers for Any-Length Video Generation Sihyun Yu Meera Hahn Dan Kondratyuk Jinwoo Shin Agrim Gupta José Lezama Irfan Essa David A. Ross Jonathan Huang DiffM VGen 77 0 0 18 Feb 2025
Can LLMs Maintain Fundamental Abilities under KV Cache Compression? Xiang Liu Zhenheng Tang Hong Chen Peijie Dong Zeyu Li Xiuze Zhou Bo Li Xuming Hu Xiaowen Chu 227 3 0 04 Feb 2025
Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment Gregor Bachmann Sotiris Anagnostidis Albert Pumarola Markos Georgopoulos A. Sanakoyeu Yuming Du Edgar Schönfeld Ali K. Thabet Jonas Kohler ALM BDL 106 8 0 31 Jan 2025
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies Nadav Timor Jonathan Mamou Daniel Korat Moshe Berchansky Oren Pereg Gaurav Jain Roy Schwartz Moshe Wasserblat David Harel 91 2 0 31 Jan 2025
Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference Zhihang Lin Mingbao Lin Luxi Lin Rongrong Ji 61 16 0 28 Jan 2025
DeServe: Towards Affordable Offline LLM Inference via Decentralization Linyu Wu Xiaoyuan Liu Tianneng Shi Zhe Ye D. Song OffRL 44 0 0 28 Jan 2025
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun Penghan Wang Fan Lai 202 1 0 15 Jan 2025
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions Doaa Mahmud Hadeel Hajmohamed Shamma Almentheri Shamma Alqaydi Lameya Aldhaheri R. A. Khalil Nasir Saeed AI4TS 51 5 0 08 Jan 2025
Unifying KV Cache Compression for Large Language Models with LeanKV Yanqi Zhang Yuwei Hu Runyuan Zhao John C. S. Lui Haibo Chen MQ 151 5 0 04 Dec 2024
Context Parallelism for Scalable Million-Token Inference Amy Yang Jingyi Yang Aya Ibrahim Xinfeng Xie Bangsheng Tang Grigory Sizov Jeremy Reizenstein Jongsoo Park Jianyu Huang MoE LRM 72 5 0 04 Nov 2024
Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRA Sangmin Bae Adam Fisch Hrayr Harutyunyan Ziwei Ji Seungyeon Kim Tal Schuster KELM 84 5 0 28 Oct 2024
EPIC: Efficient Position-Independent Context Caching for Serving Large Language Models Junhao Hu Wenrui Huang Haoran Wang Weidong Wang Tiancheng Hu Qin Zhang Hao Feng Xusheng Chen Yizhou Shan Tao Xie RALM LLMAG 42 4 0 20 Oct 2024
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration Heming Xia Yongqi Li Jun Zhang Cunxiao Du Wenjie Li LRM 56 6 0 09 Oct 2024
ControlAR: Controllable Image Generation with Autoregressive Models Zongming Li Tianheng Cheng Shoufa Chen Peize Sun Haocheng Shen Longjin Ran Xiaoxin Chen Wenyu Liu Xinggang Wang DiffM 136 15 0 03 Oct 2024
Discrete Copula Diffusion Guy Van den Broeck Oliver Broadrick Mathias Niepert Mathias Niepert DiffM 61 4 0 02 Oct 2024
Confidential Prompting: Protecting User Prompts from Cloud LLM Providers In Gim Caihua Li Lin Zhong 52 2 0 27 Sep 2024
GraphEx: A Graph-based Extraction Method for Advertiser Keyphrase Recommendation Ashirbad Mishra Soumik Dey Marshall Wu Jinyu Zhao He Yu Kaichen Ni Binbin Li Kamesh Madduri 57 1 0 05 Sep 2024
LazyLLM: Dynamic Token Pruning for Efficient Long Context LLM Inference Qichen Fu Minsik Cho Thomas Merth Sachin Mehta Mohammad Rastegari Mahyar Najibi 52 26 0 19 Jul 2024
Let the Code LLM Edit Itself When You Edit the Code Zhenyu He Jun Zhang Shengjie Luo Jingjing Xu Z. Zhang Di He KELM 39 0 0 03 Jul 2024
OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure Jikai Wang Yi Su Juntao Li Qingrong Xia Zi Ye Xinyu Duan Zhefeng Wang Min Zhang 46 13 0 25 Jun 2024
Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers Rya Sanovar Srikant Bharadwaj Renée St. Amant Victor Rühle Saravan Rajmohan 61 6 0 17 May 2024
SambaNova SN40L: Scaling the AI Memory Wall with Dataflow and Composition of Experts R. Prabhakar R. Sivaramakrishnan Darshan Gandhi Yun Du Mingran Wang ... Urmish Thakker Dawei Huang Sumti Jairath Kevin J. Brown K. Olukotun MoE 39 12 0 13 May 2024
Allo: A Programming Model for Composable Accelerator Design Hongzheng Chen Niansong Zhang Shaojie Xiang Zhichen Zeng Mengjia Dai Zhiru Zhang 54 14 0 07 Apr 2024
FastDecode: High-Throughput GPU-Efficient LLM Serving using Heterogeneous Pipelines Jiaao He Jidong Zhai 45 27 0 18 Mar 2024
GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM Hao Kang Qingru Zhang Souvik Kundu Geonhwa Jeong Zaoxing Liu Tushar Krishna Tuo Zhao MQ 43 81 0 08 Mar 2024
Yi: Open Foundation Models by 01.AI 01. AI Alex Young 01.AI Alex Young Bei Chen Chao Li ... Yue Wang Yuxuan Cai Zhenyu Gu Zhiyuan Liu Zonghong Dai OSLM LRM 150 511 0 07 Mar 2024
ReLU $^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs Zhengyan Zhang Yixin Song Guanghui Yu Xu Han Yankai Lin Chaojun Xiao Chenyang Song Zhiyuan Liu Zeyu Mi Maosong Sun 22 31 0 06 Feb 2024
Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads Tianle Cai Yuhong Li Zhengyang Geng Hongwu Peng Jason D. Lee De-huai Chen Tri Dao 60 252 0 19 Jan 2024
EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism Yanxi Chen Xuchen Pan Yaliang Li Bolin Ding Jingren Zhou LRM 41 31 0 08 Dec 2023
Splitwise: Efficient generative LLM inference using phase splitting Pratyush Patel Esha Choukse Chaojie Zhang Aashaka Shah Íñigo Goiri Saeed Maleki Ricardo Bianchini 52 204 0 30 Nov 2023
PrivateLoRA For Efficient Privacy Preserving LLM Yiming Wang Yu Lin Xiaodong Zeng Guannan Zhang 66 11 0 23 Nov 2023
Reinforcement Learning for Generative AI: A Survey Yuanjiang Cao Quan.Z Sheng Julian McAuley Lina Yao SyDa 53 10 0 28 Aug 2023
RecycleGPT: An Autoregressive Language Model with Recyclable Module Yu Jiang Qiaozhi He Xiaomin Zhuang Zhihua Wu Kunpeng Wang Wenlai Zhao Guangwen Yang KELM 28 3 0 07 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 126 11,099 0 18 Jul 2023
PowerFusion: A Tensor Compiler with Explicit Data Movement Description and Instruction-level Graph IR Zixuan Ma Haojie Wang Jingze Xing Liyan Zheng Chen Zhang Huanqi Cao Kezhao Huang Shizhi Tang Penghan Wang Jidong Zhai GNN 34 1 0 11 Jul 2023
$S$^{3}$: Increasing GPU Utilization during Generative Inference for Higher Throughput$ S $^{3}$ : Increasing GPU Utilization during Generative Inference for Higher Throughput Yunho Jin Chun-Feng Wu David Brooks Gu-Yeon Wei 34 62 0 09 Jun 2023
Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Time Zichang Liu Aditya Desai Fangshuo Liao Weitao Wang Victor Xie Zhaozhuo Xu Anastasios Kyrillidis Anshumali Shrivastava 28 202 0 26 May 2023
Dynamic Context Pruning for Efficient and Interpretable Autoregressive Transformers Sotiris Anagnostidis Dario Pavllo Luca Biggio Lorenzo Noci Aurelien Lucchi Thomas Hofmann 42 53 0 25 May 2023
Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline Zangwei Zheng Xiaozhe Ren Fuzhao Xue Yang Luo Xin Jiang Yang You 42 55 0 22 May 2023
XTREME-UP: A User-Centric Scarce-Data Benchmark for Under-Represented Languages Sebastian Ruder J. Clark Alexander Gutkin Mihir Kale Min Ma ... Dan Garrette R. Ingle Melvin Johnson Dmitry Panteleev Partha P. Talukdar ELM 26 38 0 19 May 2023