Extending Context Window of Large Language Models via Positional Interpolation

27 June 2023

Papers citing "Extending Context Window of Large Language Models via Positional Interpolation"

50 / 84 papers shown

Title
Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free Zihan Qiu Zhaoxiang Wang Bo Zheng Zeyu Huang Kaiyue Wen ... Fei Huang Suozhi Huang Dayiheng Liu Jingren Zhou Junyang Lin MoE 28 0 0 10 May 2025
The Illusion of Role Separation: Hidden Shortcuts in LLM Role Learning (and How to Fix Them) Zihao Wang Yibo Jiang Jiahao Yu Heqing Huang 38 0 0 01 May 2025
Phi-4-reasoning Technical Report Marah Abdin Sahaj Agarwal Ahmed Hassan Awadallah Vidhisha Balachandran Harkirat Singh Behl ... Vaishnavi Shrivastava Vibhav Vineet Yue Wu Safoora Yousefi Guoqing Zheng ReLM LRM 90 1 0 30 Apr 2025
Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Yixin Cao Shibo Hong Xuzhao Li Jiahao Ying Yubo Ma ... Juanzi Li Aixin Sun Xuanjing Huang Tat-Seng Chua Tianwei Zhang ALM ELM 86 2 0 26 Apr 2025
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation Yi Lu Wanxu Zhao Xin Zhou Chenxin An Cong Wang ... Jun Zhao Tao Ji Tao Gui Qi Zhang Xuanjing Huang 39 0 0 26 Apr 2025
Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts Yifei Yu Qian Zhang Lingfeng Qiao Di Yin Fang Li Jie Wang Z. Chen Suncong Zheng Xiaolong Liang Xingchen Sun 41 0 0 07 Apr 2025
Leveraging State Space Models in Long Range Genomics Matvei Popov Aymen Kallala Anirudha Ramesh Narimane Hennouni Shivesh Khaitan Rick Gentry Alain-Sam Cohen Mamba 62 0 0 07 Apr 2025
Long-Context Autoregressive Video Modeling with Next-Frame Prediction Yuchao Gu Weijia Mao Mike Zheng Shou VGen 84 2 0 25 Mar 2025
InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models Yuchen Yan Yongliang Shen Yuhang Liu Jin Jiang Hao Fei Jian Shao Yueting Zhuang LRM ReLM 53 3 0 09 Mar 2025
LongAttn: Selecting Long-context Training Data via Token-level Attention Longyun Wu Dawei Zhu Guangxiang Zhao Zhuocheng Yu Junfeng Ran Xiangyu Wong Lin Sun Sujian Li 45 0 0 24 Feb 2025
LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation Zican Dong Junyi Li Jinhao Jiang Mingyu Xu Wayne Xin Zhao Bin Wang Xin Wu VLM 213 4 0 20 Feb 2025
HeadInfer: Memory-Efficient LLM Inference by Head-wise Offloading Cheng Luo Zefan Cai Hanshi Sun Jinqi Xiao Bo Yuan Wen Xiao Junjie Hu Jiawei Zhao Beidi Chen Anima Anandkumar 69 1 0 18 Feb 2025
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs Sumin An Junyoung Sung Wonpyo Park Chanjun Park Paul Hongsuck Seo 100 0 0 10 Feb 2025
Large Language Models for In-File Vulnerability Localization Can Be "Lost in the End" Francesco Sovrano Adam Bauer Alberto Bacchelli 54 0 0 09 Feb 2025
Context-Aware Hierarchical Merging for Long Document Summarization Litu Ou Mirella Lapata MoMe 224 1 0 03 Feb 2025
SEAL: Scaling to Emphasize Attention for Long-Context Retrieval Changhun Lee Jun-gyu Jin Younghyun Cho Eunhyeok Park LRM 56 0 0 28 Jan 2025
LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion Zhan Ling Kang Liu Kai Yan Yuqing Yang Weijian Lin Ting-Han Fan Lingfeng Shen Zhengyin Du Jiecao Chen ReLM ELM LRM 52 3 0 25 Jan 2025
Fast3R: Towards 3D Reconstruction of 1000+ Images in One Forward Pass Jianing Yang Alexander Sax Kevin J Liang Mikael Henaff Hao Tang Ang Cao J. Chai Franziska Meier Matt Feiszli 3DGS 73 16 0 23 Jan 2025
ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models Thibaut Thonet Jos Rozen Laurent Besacier RALM 137 2 0 20 Jan 2025
Visual RAG: Expanding MLLM visual knowledge without fine-tuning Mirco Bonomo Simone Bianco VLM 73 5 0 18 Jan 2025
Guiding Retrieval using LLM-based Listwise Rankers Mandeep Rathee Sean MacAvaney Avishek Anand KELM LRM 65 4 0 17 Jan 2025
ListConRanker: A Contrastive Text Reranker with Listwise Encoding Junlong Liu Yue Ma Ruihui Zhao Junhao Zheng Qianli Ma Yangyang Kang 46 0 0 13 Jan 2025
Lost-in-Distance: Impact of Contextual Proximity on LLM Performance in Graph Tasks Hamed Firooz Maziar Sanjabi Wenlong Jiang Xiaoling Zhai 68 3 0 03 Jan 2025
Investigating Length Issues in Document-level Machine Translation Ziqian Peng Rachel Bawden François Yvon 69 1 0 23 Dec 2024
Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRs Michael Wornow Suhana Bedi Miguel Angel Fuentes Hernandez E. Steinberg Jason Alan Fries Christopher Ré Sanmi Koyejo N. Shah 95 4 0 09 Dec 2024
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Zehua Wang Hui Xiong Hui Xiong LLMAG 36 5 0 05 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 69 6 0 31 Oct 2024
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage Taewhoo Lee Chanwoong Yoon Kyochul Jang Donghyeon Lee Minju Song Hyunjae Kim Jaewoo Kang ELM 35 1 0 22 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 167 0 0 17 Oct 2024
Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free Ziyue Li Dinesh Manocha MoE 74 5 0 14 Oct 2024
TULIP: Token-length Upgraded CLIP Ivona Najdenkoska Mohammad Mahdi Derakhshani Yuki M. Asano Nanne van Noord Marcel Worring Cees G. M. Snoek VLM 48 3 0 13 Oct 2024
SEGMENT+: Long Text Processing with Short-Context Language Models Wei Shi Shuang Li Kerun Yu Jinglei Chen Zujie Liang ... Feng Wei Bo Zheng Jiaqing Liang Jiangjie Chen Yanghua Xiao RALM VLM 57 2 0 09 Oct 2024
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks M. Bueno R. Lotufo Rodrigo Nogueira LRM 31 0 0 08 Oct 2024
LongGenBench: Long-context Generation Benchmark Xiang Liu Peijie Dong Xuming Hu Xiaowen Chu RALM 55 8 0 05 Oct 2024
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Howard Yen Tianyu Gao Minmin Hou Ke Ding Daniel Fleischer Peter Izsak Moshe Wasserblat Danqi Chen ALM ELM 62 25 0 03 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 72 38 0 03 Oct 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 72 1 0 16 Sep 2024
You Only Use Reactive Attention Slice For Long Context Retrieval Yun Joon Soh Hanxian Huang Yuandong Tian Jishen Zhao RALM 46 0 0 03 Sep 2024
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities Peng Xu Ming-Yu Liu Xianchao Wu Zihan Liu M. Shoeybi Mohammad Shoeybi Bryan Catanzaro RALM 52 14 0 19 Jul 2024
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models Jinliang Lu Ziliang Pang Min Xiao Yaochen Zhu Rui Xia Jiajun Zhang MoMe 52 18 0 08 Jul 2024
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA Minzheng Wang Longze Chen Cheng Fu Shengyi Liao Xinghua Zhang ... Run Luo Yunshui Li Min Yang Fei Huang Yongbin Li RALM 54 44 0 25 Jun 2024
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs Ziyan Jiang Xueguang Ma Wenhu Chen RALM 55 47 0 21 Jun 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 51 12 0 20 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 77 13 0 20 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 74 56 0 11 Jun 2024
Evaluating Zero-Shot Long-Context LLM Compression Chenyu Wang Yihan Wang Kai Li 51 0 0 10 Jun 2024
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks Yusen Zhang Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang Sercan Ö. Arik RALM AI4CE LLMAG 54 28 0 04 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 48 9 0 04 Jun 2024
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models Longze Chen Ziqiang Liu Wanwei He Yunshui Li Run Luo Min Yang 42 9 0 28 May 2024
TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations Zhengwu Zhang Yuntong Hu Bo Pan Chen Ling Liang Zhao 41 2 0 27 May 2024