Efficient Streaming Language Models with Attention Sinks

29 September 2023

Michel Lang

Yuandong Tian

Beidi Chen

Song Han

Mike Lewis

AI4TS

RALM

ArXiv PDF HTML

Papers citing "Efficient Streaming Language Models with Attention Sinks"

50 / 140 papers shown

Title
Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies Xiaoliang Luo Xinyi Xu Michael Ramscar Bradley C. Love 27 0 0 13 May 2025
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Zihan Qiu Zekun Wang Bo Zheng Zeyu Huang Kaiyue Wen ... Fei Huang Suozhi Huang Dayiheng Liu Jingren Zhou Junyang Lin MoE 28 0 0 10 May 2025
LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders Zheng Chai Qin Ren Xijun Xiao H. Yang Bo Han ... Xiang Sun Yaocheng Tan Peng Xu Yuchao Zheng Di Wu 46 0 0 07 May 2025
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference Y. Chen J. Zhang Baotong Lu Qianxi Zhang Chengruidong Zhang ... Chen Chen Mingxing Zhang Yuqing Yang Fan Yang Mao Yang 38 0 0 05 May 2025
Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data Zhong Guan Likang Wu Hongke Zhao Ming He Jianpin Fan GNN 37 0 0 04 May 2025
Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing Piotr Piekos Róbert Csordás Jürgen Schmidhuber MoE VLM 99 1 0 01 May 2025
FreqKV: Frequency Domain Key-Value Compression for Efficient Context Window Extension Jushi Kai Boyi Zeng Yibo Wang Haoli Bai Bo Jiang Bo Jiang Zhouhan Lin 42 0 0 01 May 2025
The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) Zihao Wang Yibo Jiang Jiahao Yu Heqing Huang 38 0 0 01 May 2025
Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions Yiming Du Wenyu Huang Danna Zheng Zhaowei Wang Sébastien Montella Mirella Lapata Kam-Fai Wong Jeff Z. Pan KELM MU 80 2 0 01 May 2025
Fast and Low-Cost Genomic Foundation Models via Outlier Removal Haozheng Luo Chenghao Qiu Maojiang Su Zhihan Zhou Zoe Mehta Guo Ye Jerry Yao-Chieh Hu Han Liu AAML 55 1 0 01 May 2025
Towards Understanding the Nature of Attention with Low-Rank Sparse Decomposition Zhengfu He Jiadong Wang Rui Lin Xuyang Ge Wentao Shu Qiong Tang Junzhe Zhang Xipeng Qiu 70 0 0 29 Apr 2025
Softpick: No Attention Sink, No Massive Activations with Rectified Softmax Zayd Muhammad Kawakibi Zuhri Erland Hilman Fuadi Alham Fikri Aji 33 0 0 29 Apr 2025
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference Zhenyu (Allen) Zhang Zechun Liu Yuandong Tian Harshit Khaitan Zhilin Wang Steven Li 57 0 0 28 Apr 2025
TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate A. Zandieh Majid Daliri Majid Hadian Vahab Mirrokni MQ 74 0 0 28 Apr 2025
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation Yi Lu Wanxu Zhao Xin Zhou Chenxin An Cong Wang ... Jun Zhao Tao Ji Tao Gui Qi Zhang Xuanjing Huang 39 0 0 26 Apr 2025
PARD: Accelerating LLM Inference with Low-Cost PARallel Draft Model Adaptation Zihao An Huajun Bai Ziqiang Liu Dong Li E. Barsoum 61 0 0 23 Apr 2025
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism Aviv Bick Eric P. Xing Albert Gu RALM 91 0 0 22 Apr 2025
KeyDiff: Key Similarity-Based KV Cache Eviction for Long-Context LLM Inference in Resource-Constrained Environments Junyoung Park Dalton Jones Matt Morse Raghavv Goel Mingu Lee Chris Lott 27 0 0 21 Apr 2025
Efficient Pretraining Length Scaling Bohong Wu Shen Yan Sijun Zhang Jianqiao Lu Yutao Zeng Ya Wang Xun Zhou 135 0 0 21 Apr 2025
Mimic In-Context Learning for Multimodal Tasks Yuchu Jiang Jiale Fu Chenduo Hao Xinting Hu Yingzhe Peng Xin Geng Xu Yang 27 0 0 11 Apr 2025
Saliency-driven Dynamic Token Pruning for Large Language Models Yao Tao Yehui Tang Yun Wang Mingjian Zhu Hailin Hu Yunhe Wang 36 0 0 06 Apr 2025
Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models Yuheng Wu Wentao Guo Zirui Liu Heng Ji Zhaozhuo Xu Denghui Zhang 33 0 0 05 Apr 2025
Using Attention Sinks to Identify and Evaluate Dormant Heads in Pretrained LLMs Pedro Sandoval-Segura Xijun Wang Ashwinee Panda Micah Goldblum Ronen Basri Tom Goldstein David Jacobs 22 0 0 04 Apr 2025
Cognitive Memory in Large Language Models Lianlei Shan Shixian Luo Zezhou Zhu Yu Yuan Yong Wu LLMAG KELM 160 1 0 03 Apr 2025
FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning Hang Guo Yawei Li Taolin Zhang Jiadong Wang Tao Dai Shu-Tao Xia Luca Benini 72 2 0 30 Mar 2025
WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference Youhui Zuo Sibo Wei C. Zhang Zhuorui Liu Wenpeng Lu Dawei Song VLM 56 0 0 23 Mar 2025
GPU-Accelerated Motion Planning of an Underactuated Forestry Crane in Cluttered Environments M. Vu Gerald Ebmer Alexander Watcher Marc-Philip Ecker Giang Nguyen Tobias Glueck 74 2 0 18 Mar 2025
X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression Guihong Li Mehdi Rezagholizadeh Mingyu Yang Vikram Appia Emad Barsoum VLM 57 0 0 14 Mar 2025
Cost-Optimal Grouped-Query Attention for Long-Context Modeling Y. Chen Yutong Wu Chenyang Song Zhiyuan Liu Maosong Sun Xu Han Zhiyuan Liu Maosong Sun 61 0 0 12 Mar 2025
VideoScan: Enabling Efficient Streaming Video Understanding via Frame-level Semantic Carriers Ruanjun Li Yuedong Tan Yuanming Shi Jiawei Shao VLM 138 0 0 12 Mar 2025
Unleashing the Potential of Large Language Models for Text-to-Image Generation through Autoregressive Representation Alignment Xing Xie Jiawei Liu Ziyue Lin Huijie Fan Zhi Han Yandong Tang Liangqiong Qu 47 0 0 10 Mar 2025
InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models Yuchen Yan Yongliang Shen Y. Liu Jin Jiang M. Zhang Jian Shao Yueting Zhuang LRM ReLM 53 3 0 09 Mar 2025
Predicting Team Performance from Communications in Simulated Search-and-Rescue Ali Jalal-Kamali Nikolos Gurney David Pynadath AI4TS 116 8 0 05 Mar 2025
Streaming Video Question-Answering with In-context Video KV-Cache Retrieval Shangzhe Di Zhelun Yu Guanghao Zhang Haoyuan Li Tao Zhong Hao Cheng Bolin Li Wanggui He Fangxun Shu Hao Jiang 76 4 0 01 Mar 2025
MEDA: Dynamic KV Cache Allocation for Efficient Multimodal Long-Context Inference Zhongwei Wan H. Shen Xin Wang Junfeng Fang Zheda Mai M. Zhang VLM 65 3 0 24 Feb 2025
Selective Prompt Anchoring for Code Generation Yuan Tian Tianyi Zhang 94 3 0 24 Feb 2025
Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probing Qi Le Enmao Diao Ziyan Wang Xinran Wang Jie Ding Li Yang Ali Anwar 77 1 0 24 Feb 2025
LongAttn: Selecting Long-context Training Data via Token-level Attention Longyun Wu Dawei Zhu Guangxiang Zhao Zhuocheng Yu Junfeng Ran Xiangyu Wong Lin Sun Sujian Li 43 0 0 24 Feb 2025
A generative approach to LLM harmfulness detection with special red flag tokens Sophie Xhonneux David Dobre Mehrnaz Mohfakhami Leo Schwinn Gauthier Gidel 55 1 0 22 Feb 2025
Neural Attention Search Difan Deng Marius Lindauer 93 0 0 21 Feb 2025
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation Zican Dong Junyi Li Jinhao Jiang Mingyu Xu Wayne Xin Zhao Bin Wang Xin Wu VLM 213 3 0 20 Feb 2025
FairKV: Balancing Per-Head KV Cache for Fast Multi-GPU Inference Bingzhe Zhao Ke Cheng Aomufei Yuan Yuxuan Tian Ruiguang Zhong Chengchen Hu Tong Yang Lian Yu 51 0 0 19 Feb 2025
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading Cheng Luo Zefan Cai Hanshi Sun Jinqi Xiao Bo Yuan Wen Xiao Junjie Hu Jiawei Zhao Beidi Chen Anima Anandkumar 69 1 0 18 Feb 2025
Flaming-hot Initiation with Regular Execution Sampling for Large Language Models Weizhe Chen Zhicheng Zhang Guanlin Liu Renjie Zheng Wenlei Shi Chen Dun Zheng Wu Xing Jin Lin Yan ALM LRM 51 1 0 17 Feb 2025
Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs Kan Zhu Tian Tang Qinyu Xu Yile Gu Zhichen Zeng Rohan Kadekodi Liangyu Zhao Ang Li Arvind Krishnamurthy Baris Kasikci 59 2 0 17 Feb 2025
Associative Recurrent Memory Transformer Ivan Rodkin Yuri Kuratov Aydar Bulatov Andrey Kravchenko 68 2 0 17 Feb 2025
Efficient Long-Decoding Inference with Reasoning-Aware Attention Sparsity Junhao Hu Wenrui Huang Weidong Wang Zhenwen Li Tiancheng Hu Zhixia Liu Xusheng Chen Tao Xie Yizhou Shan LRM 51 0 0 16 Feb 2025
InfiniteHiP: Extending Language Model Context Up to 3 Million Tokens on a Single GPU Heejun Lee G. Park Jaduk Suh Sung Ju Hwang 87 2 0 13 Feb 2025
MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections Da Xiao Qingye Meng Shengping Li Xingyuan Yuan MoE AI4CE 66 1 0 13 Feb 2025
Efficient-vDiT: Efficient Video Diffusion Transformers With Attention Tile Hangliang Ding Dacheng Li Runlong Su Peiyuan Zhang Zhijie Deng Ion Stoica Hao Zhang VGen 73 4 0 10 Feb 2025