Extending Context Window of Large Language Models via Positional Interpolation

27 June 2023

Papers citing "Extending Context Window of Large Language Models via Positional Interpolation"

50 / 388 papers shown

Title
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks M. Bueno R. Lotufo Rodrigo Nogueira LRM 33 0 0 08 Oct 2024
Automatic Summarization of Long Documents Naman Chhibbar Jugal Kalita 31 0 0 08 Oct 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 36 3 0 07 Oct 2024
GARLIC: LLM-Guided Dynamic Progress Control with Hierarchical Weighted Graph for Long Document QA Xinyu Wang Yanzheng Xiang Lin Gui Yulan He 31 2 0 07 Oct 2024
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models Xinyu Liu Runsong Zhao Pengcheng Huang Chunyang Xiao Bei Li Jingang Wang Tong Xiao Jingbo Zhu 30 0 0 07 Oct 2024
Accelerating Inference of Networks in the Frequency Domain Chenqiu Zhao Guanfang Dong Anup Basu 48 0 0 06 Oct 2024
Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension Ning Wang Zekun Li Tongxin Bai Guoqi Li 32 0 0 05 Oct 2024
LongGenBench: Long-context Generation Benchmark Xiang Liu Peijie Dong Xuming Hu Xiaowen Chu RALM 55 8 0 05 Oct 2024
JumpStarter: Getting Started on Personal Goals with AI-Powered Context Curation Sitong Wang Xuanming Zhang Jenny Ma Alyssa Hwang Lydia B. Chilton 35 1 0 04 Oct 2024
ALR $^2$ : A Retrieve-then-Reason Framework for Long-context Question Answering Huayang Li Pat Verga Priyanka Sen Bowen Yang Vijay Viswanathan Patrick Lewis Taro Watanabe Yixuan Su RALM LRM 51 8 0 04 Oct 2024
MELODI: Exploring Memory Compression for Long Contexts Yinpeng Chen DeLesley Hutchins Aren Jansen Andrey Zhmoginov David Racz Jesper Andersen 38 2 0 04 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 72 39 0 03 Oct 2024
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Howard Yen Tianyu Gao Minmin Hou Ke Ding Daniel Fleischer Peter Izsak Moshe Wasserblat Danqi Chen ALM ELM 67 27 0 03 Oct 2024
Auto-Demo Prompting: Leveraging Generated Outputs as Demonstrations for Enhanced Batch Prompting Longyu Feng Mengze Hong Chen Jason Zhang 47 2 0 02 Oct 2024
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs Minsoo Kim Kyuhong Shim Jungwook Choi Simyung Chang 21 5 0 02 Oct 2024
Extending Context Window of Large Language Models from a Distributional Perspective Yingsheng Wu Yuxuan Gu Xiaocheng Feng Weihong Zhong Dongliang Xu Qing Yang Hongtao Liu Bing Qin 29 1 0 02 Oct 2024
Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models David Castillo-Bolado Joseph Davidson Finlay Gray Marek Rosa 34 3 0 30 Sep 2024
Visual Context Window Extension: A New Perspective for Long Video Understanding Hongchen Wei Zhenzhong Chen VLM 34 6 0 30 Sep 2024
Quality Matters: Evaluating Synthetic Data for Tool-Using LLMs Shadi Iskander Nachshon Cohen Zohar Karnin Ori Shapira Sofia Tolmach SyDa 29 1 0 24 Sep 2024
CSPS: A Communication-Efficient Sequence-Parallelism based Serving System for Transformer based Models with Long Prompts Zeyu Zhang Haiying Shen VLM 32 0 0 23 Sep 2024
ToolPlanner: A Tool Augmented LLM for Multi Granularity Instructions with Path Planning and Feedback Qinzhuo Wu Wei Liu Jian Luan Bin Wang 55 5 0 23 Sep 2024
More Effective LLM Compressed Tokens with Uniformly Spread Position Identifiers and Compression Loss Runsong Zhao Pengcheng Huang Xinyu Liu Chunyang Xiao Tong Xiao Jingbo Zhu 23 0 0 22 Sep 2024
CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data Zhao Cheng Diane Wan Matthew Abueg Sahra Ghalebikesabi Ren Yi Eugene Bagdasarian Borja Balle S. Mellem S. O’Banion 24 3 0 20 Sep 2024
Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey Sourav Verma RALM 3DV 37 2 0 20 Sep 2024
Interpolating Video-LLMs: Toward Longer-sequence LMMs in a Training-free Manner Yuzhang Shang Bingxin Xu Weitai Kang Mu Cai Yuheng Li Zehao Wen Zhen Dong Kurt Keutzer Yong Jae Lee Yan Yan 41 7 0 19 Sep 2024
A Controlled Study on Long Context Extension and Generalization in LLMs Yi Lu Jing Nathan Yan Songlin Yang Justin T. Chiu Siyu Ren Fei Yuan Wenting Zhao Zhiyong Wu Alexander M. Rush 41 9 0 18 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 72 1 0 16 Sep 2024
Inf-MLLM: Efficient Streaming Inference of Multimodal Large Language Models on a Single GPU Zhenyu Ning Jieru Zhao Qihao Jin Wenchao Ding Minyi Guo 29 6 0 11 Sep 2024
E2LLM: Encoder Elongated Large Language Models for Long-Context Understanding and Reasoning Zihan Liao Jun Wang Hang Yu Lingxiao Wei Jianguo Li Jun Wang Wei Zhang 29 3 0 10 Sep 2024
Untie the Knots: An Efficient Data Augmentation Strategy for Long-Context Pre-Training in Language Models Junfeng Tian Da Zheng Yang Cheng Rui-cang Wang C. Zhang Debing Zhang 36 4 0 07 Sep 2024
You Only Use Reactive Attention Slice For Long Context Retrieval Yun Joon Soh Hanxian Huang Yuandong Tian Jishen Zhao RALM 48 0 0 03 Sep 2024
In Defense of RAG in the Era of Long-Context Language Models Tan Yu Anbang Xu Rama Akkiraju RALM 3DV 29 24 0 03 Sep 2024
LongRecipe: Recipe for Efficient Long Context Generalization in Large Language Models Zhiyuan Hu Yuliang Liu Jinman Zhao Suyuchen Wang Yan Wang ... Qing Gu Anh Tuan Luu See-Kiong Ng Zhiwei Jiang Bryan Hooi 55 11 0 31 Aug 2024
MemLong: Memory-Augmented Retrieval for Long Text Modeling Weijie Liu Zecheng Tang Juntao Li Kehai Chen Min Zhang RALM 38 2 0 30 Aug 2024
Scaling Up Summarization: Leveraging Large Language Models for Long Text Extractive Summarization Léo Hemamou Mehdi Debiane 33 2 0 28 Aug 2024
Evidence-Enhanced Triplet Generation Framework for Hallucination Alleviation in Generative Question Answering Haowei Du Huishuai Zhang Dongyan Zhao HILM 35 0 0 27 Aug 2024
FLEURS-ASL: Including American Sign Language in Massively Multilingual Multitask Evaluation Garrett Tanzer SLR VLM 34 2 0 24 Aug 2024
Optimizing Performance: How Compact Models Match or Exceed GPT's Classification Capabilities through Fine-Tuning Baptiste Lefort Eric Benhamou Jean-Jacques Ohana David Saltiel B. Guez 30 0 0 22 Aug 2024
FocusLLM: Scaling LLM's Context by Parallel Decoding Zhenyu Li Yike Zhang Tengyu Pan Yutao Sun Zhichao Duan Junjie Fang Rong Han Zixuan Wang Jianyong Wang 42 2 0 21 Aug 2024
LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs Yushi Bai Jiajie Zhang Xin Lv Linzhi Zheng Siqi Zhu Lei Hou Yuxiao Dong Jie Tang Juanzi Li VGen LLMAG ALM 42 41 0 13 Aug 2024
AI-assisted Coding with Cody: Lessons from Context Retrieval and Evaluation for Code Recommendations Jan Hartman Rishabh Mehrotra Hitesh Sagtani Dominic Cooney Rafal Gajdulewicz Beyang Liu Julie Tibshirani Quinn Slack 41 1 0 09 Aug 2024
NACL: A General and Effective KV Cache Eviction Framework for LLMs at Inference Time Yilong Chen Guoxia Wang Junyuan Shang Shiyao Cui Zhenyu Zhang Tingwen Liu Shuohuan Wang Yu Sun Dianhai Yu Hua Wu 32 15 0 07 Aug 2024
AMES: Asymmetric and Memory-Efficient Similarity Estimation for Instance-level Retrieval Pavel Suma Giorgos Kordopatis-Zilos Ahmet Iscen Giorgos Tolias VLM 45 3 0 06 Aug 2024
Making Long-Context Language Models Better Multi-Hop Reasoners Yanyang Li Shuo Liang M. Lyu Liwei Wang LLMAG LRM 30 11 0 06 Aug 2024
DRFormer: Multi-Scale Transformer Utilizing Diverse Receptive Fields for Long Time-Series Forecasting Ruixin Ding Yuqi Chen Yu-Ting Lan Wei Zhang AI4TS 52 2 0 05 Aug 2024
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach Zhuowan Li Cheng-rong Li Mingyang Zhang Qiaozhu Mei Michael Bendersky 3DV RALM 65 38 0 23 Jul 2024
ReAttention: Training-Free Infinite Context with Finite Attention Scope Xiaoran Liu Ruixiao Li Yuerong Song Zhigeng Liu Kai Lv Hang Yan Hang Yan Linlin Li Qun Liu Xipeng Qiu LLMAG 38 1 0 21 Jul 2024
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities Peng Xu Ming-Yu Liu Xianchao Wu Zihan Liu M. Shoeybi Mohammad Shoeybi Bryan Catanzaro RALM 52 15 0 19 Jul 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 31 18 0 12 Jul 2024
Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis Jianxiang Yu Zichen Ding Jiaqi Tan Kangyang Luo Zhenmin Weng ... Chengcheng Han Qiushi Sun Zhiyong Wu Yunshi Lan Xiang Li 38 4 0 09 Jul 2024