Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models

3 February 2024

Papers citing "Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models"

33 / 33 papers shown

Title
An Empirical Study on Prompt Compression for Large Language Models Z. Zhang Jinyi Li Yihuai Lan X. Wang Hao Wang MQ 44 0 0 24 Apr 2025
FinDER: Financial Dataset for Question Answering and Evaluating Retrieval-Augmented Generation Chanyeol Choi Jihoon Kwon Jaeseon Ha Hojun Choi Chaewoon Kim Yongjae Lee Jy-yong Sohn Alejandro Lopez-Lira RALM 56 0 0 22 Apr 2025
Direct Advantage Regression: Aligning LLMs with Online AI Reward Li He He Zhao Stephen Wan Dadong Wang Lina Yao Tongliang Liu 27 0 0 19 Apr 2025
PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference Weisheng Jin Maojia Song Tej Deep Pala Yew Ken Chia Amir Zadeh Chuan Li Soujanya Poria VLM 57 0 0 30 Mar 2025
Landscape Complexity for the Empirical Risk of Generalized Linear Models: Discrimination between Structured Data Theodoros G. Tsironis Aris L. Moustakas 58 0 0 18 Mar 2025
Attention Reveals More Than Tokens: Training-Free Long-Context Reasoning with Attention-guided Retrieval Yuwei Zhang Jayanth Srinivasa Gaowen Liu Jingbo Shang LRM LLMAG RALM 87 1 0 12 Mar 2025
Attention Condensation via Sparsity Induced Regularized Training Eli Sason Darya Frolova Boris Nazarov Felix Goldberd 178 0 0 03 Mar 2025
Patient Trajectory Prediction: Integrating Clinical Notes with Transformers Sifal Klioui Sana Sellami Youssef Trardi 76 0 0 25 Feb 2025
Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding Konstantin Berestizshevsky Renzo Andri Lukas Cavigelli 80 1 0 12 Feb 2025
SelfCheckAgent: Zero-Resource Hallucination Detection in Generative Large Language Models Diyana Muhammed Gollam Rabby Sören Auer LLMAG HILM 81 0 0 03 Feb 2025
Wormhole Memory: A Rubik's Cube for Cross-Dialogue Retrieval Libo Wang 114 0 0 24 Jan 2025
Large Language Models for Constructing and Optimizing Machine Learning Workflows: A Survey Yang Gu Hengyu You Jian Cao Muran Yu Haoran Fan Shiyou Qian LM&MA AI4CE 44 3 0 11 Nov 2024
Self-Evolving Multi-Agent Collaboration Networks for Software Development Yue Hu Yuzhu Cai Yaxin Du Xinyu Zhu Xiangrui Liu Zijie Yu Yuchen Hou Shuo Tang Siheng Chen 18 10 0 22 Oct 2024
SLaNC: Static LayerNorm Calibration Mahsa Salmani Nikita Trukhanov I. Soloveychik MQ 26 0 0 14 Oct 2024
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG Bowen Jin Jinsung Yoon Jiawei Han Sercan Ö. Arik AIFin RALM 39 24 0 08 Oct 2024
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention Lijie Yang Zhihao Zhang Zhuofu Chen Zikun Li Zhihao Jia 43 4 0 07 Oct 2024
Timer-XL: Long-Context Transformers for Unified Time Series Forecasting Yong Liu Guo Qin Xiangdong Huang Jianmin Wang Mingsheng Long AI4TS 29 6 0 07 Oct 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 135 1 0 20 Sep 2024
Towards Ethical Personal AI Applications: Practical Considerations for AI Assistants with Long-Term Memory Eunhae Lee 21 0 0 17 Sep 2024
GraphInsight: Unlocking Insights in Large Language Models for Graph Structure Understanding Yukun Cao Shuo Han Zengyi Gao Zezhong Ding Xike Xie S. Kevin Zhou 37 7 0 05 Sep 2024
Learning Random Numbers to Realize Appendable Memory System for Artificial Intelligence to Acquire New Knowledge after Deployment Kazunori D Yamada 21 0 0 29 Jul 2024
Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost Sania Nayab Giulio Rossolini Giorgio Buttazzo Nicolamaria Manes F. Giacomelli Nicolamaria Manes Fabrizio Giacomelli LRM 49 24 0 29 Jul 2024
Q-S5: Towards Quantized State Space Models Steven Abreu Jens Egholm Pedersen Kade Heckel Alessandro Pierro MQ 34 7 0 13 Jun 2024
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving Pai Zeng Zhenyu Ning Jieru Zhao Weihao Cui Mengwei Xu Liwei Guo Xusheng Chen Yizhou Shan LLMAG 40 4 0 18 May 2024
A Survey on the Memory Mechanism of Large Language Model based Agents Zeyu Zhang Xiaohe Bo Chen Ma Rui Li Xu Chen Quanyu Dai Jieming Zhu Zhenhua Dong Ji-Rong Wen LLMAG KELM 39 106 0 21 Apr 2024
Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length Xuezhe Ma Xiaomeng Yang Wenhan Xiong Beidi Chen Lili Yu Hao Zhang Jonathan May Luke Zettlemoyer Omer Levy Chunting Zhou 53 27 0 12 Apr 2024
LLM in a flash: Efficient Large Language Model Inference with Limited Memory Keivan Alizadeh-Vahid Iman Mirzadeh Dmitry Belenko Karen Khatamifard Minsik Cho C. C. D. Mundo Mohammad Rastegari Mehrdad Farajtabar 72 112 0 12 Dec 2023
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models Wai-Chung Kwan Xingshan Zeng Yufei Wang Yusen Sun Liangyou Li Lifeng Shang Qun Liu Kam-Fai Wong ELM 89 10 0 30 Oct 2023
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression Huiqiang Jiang Qianhui Wu Xufang Luo Dongsheng Li Chin-Yew Lin Yuqing Yang Lili Qiu RALM 112 183 0 10 Oct 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 266 0 11 Mar 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 250 695 0 27 Aug 2021
A Simple and Effective Positional Encoding for Transformers Pu-Chin Chen Henry Tsai Srinadh Bhojanapalli Hyung Won Chung Yin-Wen Chang Chun-Sung Ferng 59 62 0 18 Apr 2021
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 243 580 0 12 Mar 2020