Extending Context Window of Large Language Models via Positional Interpolation

27 June 2023

Papers citing "Extending Context Window of Large Language Models via Positional Interpolation"

50 / 388 papers shown

Title
Toward Conversational Agents with Context and Time Sensitive Long-term Memory Nick Alonso Tomás Figliolia A. Ndirango Beren Millidge RALM 3DV 61 3 0 29 May 2024
Long Context is Not Long at All: A Prospector of Long-Dependency Data for Large Language Models Longze Chen Ziqiang Liu Wanwei He Yunshui Li Run Luo Min Yang 42 9 0 28 May 2024
XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference Shengnan Wang Youhui Bai Lin Zhang Pingyi Zhou Shixiong Zhao Gong Zhang Sen Wang Renhai Chen Hua Xu Hongwei Sun 36 3 0 28 May 2024
TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations Zhengwu Zhang Yuntong Hu Bo Pan Chen Ling Liang Zhao 46 2 0 27 May 2024
Compressing Lengthy Context With UltraGist Peitian Zhang Zheng Liu Shitao Xiao Ninglu Shao Qiwei Ye Zhicheng Dou 40 4 0 26 May 2024
Base of RoPE Bounds Context Length Xin Men Mingyu Xu Bingning Wang Qingyu Zhang Hongyu Lin Xianpei Han Weipeng Chen 42 20 0 23 May 2024
The CAP Principle for LLM Serving: A Survey of Long-Context Large Language Model Serving Pai Zeng Zhenyu Ning Jieru Zhao Weihao Cui Mengwei Xu Liwei Guo Xusheng Chen Yizhou Shan LLMAG 48 4 0 18 May 2024
RoTHP: Rotary Position Embedding-based Transformer Hawkes Process Anningzhe Gao Shan Dai 31 3 0 11 May 2024
Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding? Yutong Hu Quzhe Huang Mingxu Tao Chen Zhang Yansong Feng 48 23 0 09 May 2024
Lumina-T2X: Transforming Text into Any Modality, Resolution, and Duration via Flow-based Large Diffusion Transformers Peng Gao Le Zhuo Ziyi Lin Ruoyi Du Xu Luo ... Weicai Ye He Tong Jingwen He Yu Qiao Hongsheng Li VGen 37 84 0 09 May 2024
Long Context Alignment with Short Instructions and Synthesized Positions Wenhao Wu Yizhong Wang Yao Fu Xiang Yue Dawei Zhu Sujian Li SyDa 54 18 0 07 May 2024
Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation Kaize Shi Xueyao Sun Qing Li Guandong Xu 56 13 0 06 May 2024
Extending Llama-3's Context Ten-Fold Overnight Peitian Zhang Ninglu Shao Zheng Liu Shitao Xiao Hongjin Qian Qiwei Ye Zhicheng Dou SyDa 42 14 0 30 Apr 2024
In-Context Learning with Long-Context Models: An In-Depth Exploration Amanda Bertsch Maor Ivgi Uri Alon Jonathan Berant Matthew R. Gormley Matthew R. Gormley Graham Neubig ReLM AIMat 93 65 0 30 Apr 2024
Make Your LLM Fully Utilize the Context Shengnan An Zexiong Ma Zeqi Lin Nanning Zheng Jian-Guang Lou SyDa 59 55 0 25 Apr 2024
Length Generalization of Causal Transformers without Position Encoding Jie Wang Tao Ji Yuanbin Wu Hang Yan Tao Gui Qi Zhang Xuanjing Huang Xiaoling Wang VLM 55 15 0 18 Apr 2024
LongEmbed: Extending Embedding Models for Long Context Retrieval Dawei Zhu Liang Wang Nan Yang Yifan Song Wenhao Wu Furu Wei Sujian Li RALM 47 22 0 18 Apr 2024
Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs Woomin Song Seunghyuk Oh Sangwoo Mo Jaehyung Kim Sukmin Yun Jung-Woo Ha Jinwoo Shin 43 14 0 16 Apr 2024
Leveraging Temporal Contextualization for Video Action Recognition Minji Kim Dongyoon Han Taekyung Kim Bohyung Han 55 2 0 15 Apr 2024
TransformerFAM: Feedback attention is working memory Dongseong Hwang Weiran Wang Zhuoyuan Huo K. Sim P. M. Mengibar 40 12 0 14 Apr 2024
LLoCO: Learning Long Contexts Offline Sijun Tan Xiuyu Li Shishir G. Patil Ziyang Wu Tianjun Zhang Kurt Keutzer Joseph E. Gonzalez Raluca A. Popa RALM OffRL LLMAG 46 6 0 11 Apr 2024
Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention Tsendsuren Munkhdalai Manaal Faruqui Siddharth Gopal LRM LLMAG CLL 91 106 0 10 Apr 2024
Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation Thomas Merth Qichen Fu Mohammad Rastegari Mahyar Najibi LRM RALM 39 9 0 10 Apr 2024
Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarks Chonghua Wang Haodong Duan Songyang Zhang Dahua Lin Kai-xiang Chen ELM 31 17 0 09 Apr 2024
Emergent Abilities in Reduced-Scale Generative Language Models Sherin Muckatira Vijeta Deshpande Vladislav Lialin Anna Rumshisky ReLM ELM LRM 41 4 0 02 Apr 2024
Long-context LLMs Struggle with Long In-context Learning Tianle Li Ge Zhang Quy Duc Do Xiang Yue Wenhu Chen 56 164 0 02 Apr 2024
GUARD-D-LLM: An LLM-Based Risk Assessment Engine for the Downstream uses of LLMs Sundaraparipurnan Narayanan Sandeep Vishwakarma 46 3 0 02 Apr 2024
InsightLens: Discovering and Exploring Insights from Conversational Contexts in Large-Language-Model-Powered Data Analysis Luoxuan Weng Xingbo Wang Junyu Lu Yingchaojie Feng Yihan Liu Wei Chen 60 1 0 02 Apr 2024
A Survey on Large Language Model-Based Game Agents Sihao Hu Tiansheng Huang Gaowen Liu Ramana Rao Kompella Gaowen Liu Selim Furkan Tekin Yichang Xu Zachary Yahn Ling Liu LLMAG LM&Ro AI4CE LM&MA 71 52 0 02 Apr 2024
Decoding the Digital Fine Print: Navigating the potholes in Terms of service/ use of GenAI tools against the emerging need for Transparent and Trustworthy Tech Futures Sundaraparipurnan Narayanan 34 0 0 26 Mar 2024
AIOS: LLM Agent Operating System Kai Mei Zelong Li Wujiang Xu Wenyue Hua Mingyu Jin Yongfeng Zhang Shuyuan Xu Ruosong Ye Yingqiang Ge Yongfeng Zhang LLMAG 30 17 0 25 Mar 2024
Large Language Models Offer an Alternative to the Traditional Approach of Topic Modelling Yida Mu Chun Dong Kalina Bontcheva Xingyi Song 31 19 0 24 Mar 2024
Holographic Global Convolutional Networks for Long-Range Prediction Tasks in Malware Detection Mohammad Mahmudul Alam Edward Raff Stella Biderman Tim Oates James Holt AAML 38 3 0 23 Mar 2024
A hybrid LLM workflow can help identify user privilege related variables in programs of any size Haizhou Wang Zhilong Wang Peng Liu 45 3 0 23 Mar 2024
Long-CLIP: Unlocking the Long-Text Capability of CLIP Beichen Zhang Pan Zhang Xiao-wen Dong Yuhang Zang Jiaqi Wang CLIP VLM 45 110 0 22 Mar 2024
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey Zeyu Han Chao Gao Jinyang Liu Jeff Zhang Sai Qian Zhang 150 319 0 21 Mar 2024
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models Yaowei Zheng Richong Zhang Junhao Zhang Yanhan Ye Zheyan Luo Zhangchi Feng Yongqiang Ma 55 401 0 20 Mar 2024
NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens Cunxiang Wang Ruoxi Ning Boqi Pan Tonghui Wu Qipeng Guo ... Guangsheng Bao Xiangkun Hu Zheng Zhang Qian Wang Yue Zhang RALM 106 4 0 18 Mar 2024
A Continued Pretrained LLM Approach for Automatic Medical Note Generation Dong Yuan Eti Rastogi Gautam Naik Sree Prasanna Rajagopal Sagar Goyal Fen Zhao Jai Chintagunta Jeff Ward LM&MA AI4MH 45 20 0 14 Mar 2024
StreamingDialogue: Prolonged Dialogue Learning via Long Context Compression with Minimal Losses Jia-Nan Li Quan Tu Cunli Mao Zhengtao Yu Ji-Rong Wen Rui Yan OffRL 29 3 0 13 Mar 2024
RepoHyper: Better Context Retrieval Is All You Need for Repository-Level Code Completion Huy N. Phan Hoang N. Phan Tien N. Nguyen Nghi D. Q. Bui 50 3 0 10 Mar 2024
Can't Remember Details in Long Documents? You Need Some R&R Devanshu Agrawal Shang Gao Martin Gajek RALM 200 6 0 08 Mar 2024
CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models Zexuan Qiu Jingjing Li Shijue Huang Wanjun Zhong Irwin King ELM ALM 53 3 0 06 Mar 2024
Breeze-7B Technical Report Chan-Jan Hsu Chang-Le Liu Feng-Ting Liao Po-Chun Hsu Yi-Chang Chen Da-Shan Shiu 34 2 0 05 Mar 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 34 26 0 05 Mar 2024
Rethinking LLM Language Adaptation: A Case Study on Chinese Mixtral Yiming Cui Xin Yao 30 4 0 04 Mar 2024
VisionLLaMA: A Unified LLaMA Backbone for Vision Tasks Xiangxiang Chu Jianlin Su Bo Zhang Chunhua Shen MLLM 44 10 0 01 Mar 2024
Resonance RoPE: Improving Context Length Generalization of Large Language Models Suyuchen Wang I. Kobyzev Peng Lu Mehdi Rezagholizadeh Bang Liu 43 11 0 29 Feb 2024
NextLevelBERT: Masked Language Modeling with Higher-Level Representations for Long Documents Tamara Czinczoll Christoph Hones Maximilian Schall Gerard de Melo 46 2 0 27 Feb 2024
Training-Free Long-Context Scaling of Large Language Models Chen An Fei Huang Jun Zhang Shansan Gong Xipeng Qiu Chang Zhou Lingpeng Kong ALM LRM 40 35 0 27 Feb 2024