Extending Context Window of Large Language Models via Positional Interpolation

27 June 2023

Papers citing "Extending Context Window of Large Language Models via Positional Interpolation"

50 / 388 papers shown

Title
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models Jinliang Lu Ziliang Pang Min Xiao Yaochen Zhu Rui Xia Jiajun Zhang MoMe 59 18 0 08 Jul 2024
On the Power of Convolution Augmented Transformer Mingchen Li Xuechen Zhang Yixiao Huang Samet Oymak 40 1 0 08 Jul 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Huiqiang Jiang Yucheng Li Chengruidong Zhang Qianhui Wu Xufang Luo ... Amir H. Abdi Dongsheng Li Chin-Yew Lin Yuqing Yang L. Qiu 72 87 0 02 Jul 2024
Neurocache: Efficient Vector Retrieval for Long-range Language Modeling Ali Safaya Deniz Yuret 42 1 0 02 Jul 2024
Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models Weihong Zhong Xiaocheng Feng Liang Zhao Qiming Li Lei Huang Yuxuan Gu Weitao Ma Yuan Xu Bing Qin MLLM 41 10 0 30 Jun 2024
InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Management Wonbeom Lee Jungi Lee Junghwan Seo Jaewoong Sim RALM 34 75 0 28 Jun 2024
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness Hongzhan Lin Ang Lv Yuhan Chen Chen Zhu Yang Song Hengshu Zhu Rui Yan 47 9 0 28 Jun 2024
From Artificial Needles to Real Haystacks: Improving Retrieval Capabilities in LLMs by Finetuning on Synthetic Data Zheyang Xiong Vasilis Papageorgiou Kangwook Lee Dimitris Papailiopoulos SyDa RALM 41 12 0 27 Jun 2024
Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs Lei Zhang Yunshui Li Jiaming Li Xiaobo Xia Jiaxi Yang Run Luo Minzheng Wang Longze Chen Junhao Liu Min Yang 40 2 0 26 Jun 2024
MammothModa: Multi-Modal Large Language Model Qi She Junwen Pan Xin Wan Rui Zhang Dawei Lu Kai Huang MLLM VLM 41 1 0 26 Jun 2024
UIO-LLMs: Unbiased Incremental Optimization for Long-Context LLMs Wenhao Li Mingbao Lin Mingliang Xu Shuicheng Yan Rongrong Ji 43 0 0 26 Jun 2024
LongIns: A Challenging Long-context Instruction-based Exam for LLMs Shawn Gavin Tuney Zheng Jiaheng Liu Quehry Que Noah Wang Jian Yang Chenchen Zhang Wenhao Huang Wenhu Chen Ge Zhang RALM LRM 42 3 0 25 Jun 2024
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA Minzheng Wang Longze Chen Cheng Fu Shengyi Liao Xinghua Zhang ... Run Luo Yunshui Li Min Yang Fei Huang Yongbin Li RALM 54 44 0 25 Jun 2024
CLERC: A Dataset for Legal Case Retrieval and Retrieval-Augmented Analysis Generation Abe Bohan Hou Orion Weller Guanghui Qin Eugene Yang Dawn J Lawrie Nils Holzenberger Andrew Blair-Stanek Benjamin Van Durme AILaw ELM 87 6 0 24 Jun 2024
Long Context Transfer from Language to Vision Peiyuan Zhang Kaichen Zhang Bo Li Guangtao Zeng Jingkang Yang Yuanhan Zhang Ziyue Wang Haoran Tan Chunyuan Li Ziwei Liu VLM 72 146 0 24 Jun 2024
Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning Brandon Huang Chancharik Mitra Assaf Arbelle Leonid Karlinsky Trevor Darrell Roei Herzig 49 13 0 21 Jun 2024
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs Ziyan Jiang Xueguang Ma Wenhu Chen RALM 60 48 0 21 Jun 2024
MoA: Mixture of Sparse Attention for Automatic Large Language Model Compression Tianyu Fu Haofeng Huang Xuefei Ning Genghan Zhang Boju Chen ... Shiyao Li Shengen Yan Guohao Dai Huazhong Yang Yu Wang MQ 52 17 0 21 Jun 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 53 12 0 20 Jun 2024
GraphReader: Building Graph-based Agent to Enhance Long-Context Abilities of Large Language Models Shilong Li Yancheng He Hangyu Guo Xingyuan Bu Ge Bai ... Xingwei Qu Yangguang Li Wanli Ouyang Wenbo Su Bo Zheng RALM LLMAG 51 7 0 20 Jun 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 77 13 0 20 Jun 2024
Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective M. Zhong Chen Zhang Yikun Lei Xikai Liu Yan Gao Yao Hu Kehai Chen Min Zhang 45 5 0 19 Jun 2024
QRMeM: Unleash the Length Limitation through Question then Reflection Memory Mechanism Bo Wang Heyan Huang Yixin Cao Jiahao Ying Wei Tang Chong Feng KELM 45 0 0 19 Jun 2024
What Are the Odds? Language Models Are Capable of Probabilistic Reasoning Akshay Paruchuri Jake Garrison Shun Liao John Hernandez Jacob Sunshine Tim Althoff Xin Liu Daniel J. McDuff LRM 39 7 0 18 Jun 2024
ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools Team GLM : Aohan Zeng Bin Xu Bowen Wang ... Zhaoyu Wang Zhen Yang Zhengxiao Du Zhenyu Hou Zihan Wang ALM 79 515 0 18 Jun 2024
LLM4MSR: An LLM-Enhanced Paradigm for Multi-Scenario Recommendation Yuhao Wang Yichao Wang Zichuan Fu Xiangyang Li Xiangyu Zhao Huifeng Guo Ruiming Tang 46 13 0 18 Jun 2024
Retrieval Meets Reasoning: Dynamic In-Context Editing for Long-Text Understanding Weizhi Fei Xueyan Niu Guoqing Xie Yanhua Zhang Bo Bai Lei Deng Wei Han LRM KELM RALM 38 5 0 18 Jun 2024
CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling Yu Bai Xiyuan Zou Heyan Huang Sanxing Chen Marc-Antoine Rondeau Yang Gao Jackie Chi Kit Cheung 39 4 0 17 Jun 2024
SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention Qianchao Zhu Jiangfei Duan Chang Chen Siran Liu Xiuhong Li ... Huanqi Cao Xiao Chuanfu Xingcheng Zhang Dahua Lin Chao Yang 30 16 0 17 Jun 2024
What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling Yutong Hu Quzhe Huang Kangcheng Luo Yansong Feng 56 1 0 17 Jun 2024
3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding Xindian Ma Wenyuan Liu Peng Zhang Nan Xu 44 3 0 14 Jun 2024
From Text to Life: On the Reciprocal Relationship between Artificial Life and Large Language Models Eleni Nisioti Claire Glanois Elias Najarro Andrew Dai Elliot Meyerson J. Pedersen Laetitia Teodorescu Conor F. Hayes Shyam Sudhakaran Sebastian Risi AI4CE LM&Ro 56 3 0 14 Jun 2024
LieRE: Generalizing Rotary Position Encodings Sophie Ostmeier Brian Axelrod Michael E. Moseley Akshay S. Chaudhari C. Langlotz 36 0 0 14 Jun 2024
Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling Liliang Ren Yang Liu Yadong Lu Yelong Shen Chen Liang Weizhu Chen Mamba 77 57 0 11 Jun 2024
AutoSurvey: Large Language Models Can Automatically Write Surveys Yidong Wang Qi Guo Wenjin Yao Hongbo Zhang Xin Zhang ... Hao Fei Qingsong Wen Wei Ye Shikun Zhang Yue Zhang LM&MA 30 19 0 10 Jun 2024
Recurrent Context Compression: Efficiently Expanding the Context Window of LLM Chensen Huang Guibo Zhu Xuepeng Wang Yifei Luo Guojing Ge Haoran Chen Dong Yi Jinqiao Wang 67 1 0 10 Jun 2024
Evaluating Zero-Shot Long-Context LLM Compression Chenyu Wang Yihan Wang Kai Li 51 0 0 10 Jun 2024
SinkLoRA: Enhanced Efficiency and Chat Capabilities for Long-Context Large Language Models Hengyu Zhang RALM 47 2 0 09 Jun 2024
LoCoCo: Dropping In Convolutions for Long Context Compression Ruisi Cai Yuandong Tian Zhangyang Wang Beidi Chen 49 10 0 08 Jun 2024
Lumina-Next: Making Lumina-T2X Stronger and Faster with Next-DiT Le Zhuo Ruoyi Du Han Xiao Yangguang Li Dongyang Liu ... Wanli Ouyang Ziwei Liu Ping Luo Hongsheng Li Peng Gao 52 47 0 05 Jun 2024
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks Yusen Zhang Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang Sercan Ö. Arik RALM AI4CE LLMAG 59 30 0 04 Jun 2024
Leveraging Visual Tokens for Extended Text Contexts in Multi-Modal Learning Alex Jinpeng Wang Linjie Li Yiqi Lin Min Li Lijuan Wang Mike Zheng Shou VLM 33 3 0 04 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 52 9 0 04 Jun 2024
Retaining Key Information under High Compression Ratios: Query-Guided Compressor for LLMs Zhiwei Cao Qian Cao Yu Lu Ningxin Peng Luyang Huang Shanbo Cheng Jinsong Su 42 11 0 04 Jun 2024
Extended Mind Transformers Phoebe Klett Thomas Ahle RALM 29 0 0 04 Jun 2024
LongSSM: On the Length Extension of State-space Models in Language Modelling Shida Wang 48 0 0 04 Jun 2024
R2C2-Coder: Enhancing and Benchmarking Real-world Repository-level Code Completion Abilities of Code Large Language Models Ken Deng Jiaheng Liu He Zhu Congnan Liu Jingxin Li ... Yuanxing Zhang Wenbo Su Bangyu Xiang Tiezheng Ge Bo Zheng 50 2 0 03 Jun 2024
LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models Liang Zhao Tianwen Wei Liang Zeng Cheng Cheng Liu Yang ... Yimeng Gan Rui Hu Shuicheng Yan Han Fang Yahui Zhou LLMAG SyDa 66 10 0 02 Jun 2024
PostDoc: Generating Poster from a Long Multimodal Document Using Deep Submodular Optimization Vijay Jaisankar Sambaran Bandyopadhyay Kalp Vyas Varre Chaitanya Shwetha Somasundaram 32 2 0 30 May 2024
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model Chaochen Gao Xing Wu Qingfang Fu Songlin Hu SyDa 34 5 0 30 May 2024