v1v2 (latest)

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

7 November 2023

Papers citing "Prompt Cache: Modular Attention Reuse for Low-Latency Inference"

27 / 27 papers shown

Title
From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs Yaxiong Wu Sheng Liang Chen Zhang Yucheng Wang Yanzhe Zhang Huifeng Guo Ruiming Tang Yong Liu KELM 122 7 0 22 Apr 2025
KVLink: Accelerating Large Language Models via Efficient KV Cache Reuse Jingbo Yang Bairu Hou Wei Wei Yujia Bao Shiyu Chang VLM 172 3 0 21 Feb 2025
DiSCo: Device-Server Collaborative LLM-Based Text Streaming Services Ting Sun Penghan Wang Fan Lai 80 0 0 17 Feb 2025
HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location Ting Sun Penghan Wang Fan Lai 541 2 0 15 Jan 2025
FlexCache: Flexible Approximate Cache System for Video Diffusion Desen Sun Henry Tian Tim Lu Sihang Liu DiffM 138 1 0 18 Dec 2024
Marconi: Prefix Caching for the Era of Hybrid LLMs Rui Pan Zhuang Wang Zhen Jia Can Karakus Luca Zancato Tri Dao Ravi Netravali Yida Wang 171 4 0 28 Nov 2024
PyGen: A Collaborative Human-AI Approach to Python Package Creation Saikat Barua Mostafizur Rahman Md Jafor Sadek Rafiul Islam Shehnaz Khaled Md. Shohrab Hossain 115 2 0 13 Nov 2024
Geometric Collaborative Filtering with Convergence Hisham Husain Julien Monteil FedML 127 9 0 04 Oct 2024
The Early Bird Catches the Leak: Unveiling Timing Side Channels in LLM Serving Systems Linke Song Zixuan Pang Wenhao Wang Zihao Wang XiaoFeng Wang Hongbo Chen Wei Song Yier Jin Dan Meng Rui Hou 121 8 0 30 Sep 2024
Confidential Prompting: Protecting User Prompts from Cloud LLM Providers In Gim Caihua Li Lin Zhong 123 3 0 27 Sep 2024
Teola: Towards End-to-End Optimization of LLM-based Applications Xin Tan Yimin Jiang Yitao Yang Hong-Yu Xu 151 7 0 29 Jun 2024
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion Jiayi Yao Hanchen Li Yuhan Liu Siddhant Ray Yihua Cheng Qizheng Zhang Kuntai Du Shan Lu Junchen Jiang 107 24 0 26 May 2024
LongCoder: A Long-Range Pre-trained Language Model for Code Completion Daya Guo Canwen Xu Nan Duan Jian Yin Julian McAuley 68 89 0 26 Jun 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 414 2,393 0 09 Nov 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 339 775 0 27 Aug 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 329 2,533 0 20 Apr 2021
QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization Ming Zhong Da Yin Tao Yu A. Zaidi Mutethia Mutuma ... Ahmed Hassan Awadallah Asli Celikyilmaz Yang Liu Xipeng Qiu Dragomir R. Radev RALM 89 339 0 13 Apr 2021
Efficient Attentions for Long Document Summarization L. Huang Shuyang Cao Nikolaus Nova Parulian Heng Ji Lu Wang 133 289 0 05 Apr 2021
Position Information in Transformers: An Overview Philipp Dufter Martin Schmitt Hinrich Schütze 93 148 0 22 Feb 2021
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 170 1,128 0 14 Sep 2020
Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction L. Rasmy Yang Xiang Z. Xie Cui Tao Degui Zhi AI4MH LM&MA 99 696 0 22 May 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 187 4,100 0 10 Apr 2020
Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Data E. Steinberg Kenneth Jung Jason Alan Fries Conor K. Corbin Stephen Pfohl N. Shah 88 109 0 06 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 341 1,918 0 17 Sep 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 129 3,156 0 01 Apr 2019
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W. Cohen Ruslan Salakhutdinov Christopher D. Manning RALM 212 2,700 0 25 Sep 2018
The NarrativeQA Reading Comprehension Challenge Tomás Kociský Jonathan Richard Schwarz Phil Blunsom Chris Dyer Karl Moritz Hermann Gábor Melis Edward Grefenstette 142 787 0 19 Dec 2017