Extending Context Window of Large Language Models via Positional Interpolation

27 June 2023

Papers citing "Extending Context Window of Large Language Models via Positional Interpolation"

50 / 388 papers shown

Title
Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems Xupeng Miao Gabriele Oliaro Zhihao Zhang Xinhao Cheng Hongyi Jin Tianqi Chen Zhihao Jia 73 77 0 23 Dec 2023
Training With "Paraphrasing the Original Text'' Improves Long-Context Performance Yijiong Yu RALM 26 3 0 18 Dec 2023
A Survey of Reasoning with Foundation Models Jiankai Sun Chuanyang Zheng Enze Xie Zhengying Liu Ruihang Chu ... Xipeng Qiu Yi-Chen Guo Hui Xiong Qun Liu Zhenguo Li ReLM LRM AI4CE 32 79 0 17 Dec 2023
HyperPIE: Hyperparameter Information Extraction from Scientific Publications T. Saier Mayumi Ohta Takuto Asakura Michael Färber 26 4 0 17 Dec 2023
Extending Context Window of Large Language Models via Semantic Compression Weizhi Fei Xueyan Niu Pingyi Zhou Lu Hou Bo Bai Lei Deng Wei Han 46 27 0 15 Dec 2023
Marathon: A Race Through the Realm of Long Context with Large Language Models Lei Zhang Yunshui Li Ziqiang Liu Jiaxi Yang Junhao Liu Longze Chen Run Luo Min Yang OffRL LRM 45 6 0 15 Dec 2023
The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation Rongwu Xu Brian S. Lin Shujian Yang Tianqi Zhang Weiyan Shi Tianwei Zhang Zhixuan Fang Wei Xu Han Qiu 52 51 0 14 Dec 2023
Fewer is More: Boosting LLM Reasoning with Reinforced Context Pruning Xijie Huang Li Lyna Zhang Kwang-Ting Cheng Fan Yang Mao Yang LRM ReLM 34 8 0 14 Dec 2023
Zebra: Extending Context Window with Layerwise Grouped Local-Global Attention Kaiqiang Song Xiaoyang Wang Sangwoo Cho Xiaoman Pan Dong Yu 42 7 0 14 Dec 2023
SCCA: Shifted Cross Chunk Attention for long contextual semantic expansion Yuxiang Guo 20 0 0 12 Dec 2023
The Generalization Gap in Offline Reinforcement Learning Ishita Mediratta Qingfei You Minqi Jiang Roberta Raileanu OffRL 92 10 0 10 Dec 2023
Using Large Language Models for Hyperparameter Optimization Michael Ruogu Zhang Nishkrit Desai Juhan Bae Jonathan Lorraine Jimmy Ba 36 51 0 07 Dec 2023
Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use Yuhan Chen Ang Lv Ting-En Lin C. Chen Yuchuan Wu Fei Huang Yongbin Li Rui Yan 34 24 0 07 Dec 2023
LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem Yingqiang Ge Yujie Ren Wenyue Hua Shuyuan Xu Juntao Tan Yongfeng Zhang LLMAG 23 28 0 06 Dec 2023
Exchange-of-Thought: Enhancing Large Language Model Capabilities through Cross-Model Communication Zhangyue Yin Qiushi Sun Cheng Chang Qipeng Guo Junqi Dai Xuanjing Huang Xipeng Qiu LRM 56 50 0 04 Dec 2023
Clinical Risk Prediction Using Language Models: Benefits And Considerations Angeela Acharya Sulabh Shrestha Anyi Chen Joseph Conte Sanja Avramovic S. Sikdar Antonios Anastasopoulos Sanmay Das 26 7 0 29 Nov 2023
LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models Yanwei Li Chengyao Wang Jiaya Jia VLM MLLM 58 264 0 28 Nov 2023
ChatGPT's One-year Anniversary: Are Open-Source Large Language Models Catching up? Hailin Chen Fangkai Jiao Xingxuan Li Chengwei Qin Mathieu Ravaut Ruochen Zhao Caiming Xiong Chenyu You ELM CLL AI4MH LRM ALM 85 27 0 28 Nov 2023
One Pass Streaming Algorithm for Super Long Token Attention Approximation in Sublinear Space Raghav Addanki Chenyang Li Zhao Song Chiwun Yang 55 3 0 24 Nov 2023
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 49 56 0 21 Nov 2023
AcademicGPT: Empowering Academic Research Shufa Wei Xiaolong Xu Xianbiao Qi Xi Yin Jun Xia ... Chihao Dai Lihua Wang Xiaohui Liu Lei Zhang Yutao Xie LM&MA 47 3 0 21 Nov 2023
Scaling TabPFN: Sketching and Feature Selection for Tabular Prior-Data Fitted Networks Ben Feuer Chinmay Hegde Niv Cohen 42 10 0 17 Nov 2023
TableLlama: Towards Open Large Generalist Models for Tables Tianshu Zhang Xiang Yue Yifei Li Huan Sun LMTD ALM 20 83 0 15 Nov 2023
Thread of Thought Unraveling Chaotic Contexts Yucheng Zhou Xiubo Geng Tao Shen Chongyang Tao Guodong Long Jian-Guang Lou Jianbing Shen LRM 42 40 0 15 Nov 2023
LongQLoRA: Efficient and Effective Method to Extend Context Length of Large Language Models Jianxin Yang 27 6 0 08 Nov 2023
LooGLE: Can Long-Context Language Models Understand Long Contexts? Jiaqi Li Mengmeng Wang Zilong Zheng Muhan Zhang ELM RALM 40 108 0 08 Nov 2023
Human-AI Collaboration in Thematic Analysis using ChatGPT: A User Study and Design Recommendations Lixiang Yan Vanessa Echeverría G. Fernandez-Nieto Yueqiao Jin Z. Swiecki Linxuan Zhao D. Gašević Roberto Martínez-Maldonado 20 13 0 07 Nov 2023
Navigating Scaling Laws: Compute Optimality in Adaptive Model Training Sotiris Anagnostidis Gregor Bachmann Imanol Schlag Thomas Hofmann 35 2 0 06 Nov 2023
Can ChatGPT support software verification? Christian Janssen Cedric Richter Heike Wehrheim 16 10 0 04 Nov 2023
Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation Ta-Chung Chi Ting-Han Fan Alexander I. Rudnicky 22 4 0 01 Nov 2023
Skywork: A More Open Bilingual Foundation Model Tianwen Wei Liang Zhao Lichang Zhang Bo Zhu Lijie Wang ... Yongyi Peng Xiaojuan Liang Shuicheng Yan Han Fang Yahui Zhou 38 93 0 30 Oct 2023
M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models Wai-Chung Kwan Xingshan Zeng Yufei Wang Yusen Sun Liangyou Li Lifeng Shang Qun Liu Kam-Fai Wong ELM 97 10 0 30 Oct 2023
CLEX: Continuous Length Extrapolation for Large Language Models Guanzheng Chen Xin Li Zaiqiao Meng Shangsong Liang Li Bing 37 29 0 25 Oct 2023
Large Language Models are Visual Reasoning Coordinators Liangyu Chen Bo Li Sheng Shen Jingkang Yang Chunyuan Li Kurt Keutzer Trevor Darrell Ziwei Liu VLM LRM 41 51 0 23 Oct 2023
Extending Input Contexts of Language Models through Training on Segmented Sequences Petros Karypis Julian McAuley George Karypis 32 0 0 23 Oct 2023
Large Search Model: Redefining Search Stack in the Era of LLMs Liang Wang Nan Yang Xiaolong Huang Linjun Yang Rangan Majumder Furu Wei LRM KELM 47 13 0 23 Oct 2023
Position Interpolation Improves ALiBi Extrapolation Faisal Al-Khateeb Nolan Dey Daria Soboleva Joel Hestness LLMSV 34 5 0 18 Oct 2023
From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers Shaoxiong Duan Yining Shi Wei Xu 28 8 0 18 Oct 2023
In-context Pretraining: Language Modeling Beyond Document Boundaries Weijia Shi Sewon Min Maria Lomeli Chunting Zhou Margaret Li ... Victoria Lin Noah A. Smith Luke Zettlemoyer Scott Yih Mike Lewis LRM RALM SyDa 34 48 0 16 Oct 2023
On Context Utilization in Summarization with Large Language Models Mathieu Ravaut Aixin Sun Nancy F. Chen Chenyu You 44 13 0 16 Oct 2023
End-to-end Story Plot Generator Hanlin Zhu Andrew Cohen Danqing Wang Kevin Kaichuang Yang Xiaomeng Yang Jiantao Jiao Yuandong Tian 27 5 0 13 Oct 2023
MemGPT: Towards LLMs as Operating Systems Charles Packer Sarah Wooders Kevin Lin Vivian Fang Shishir G. Patil Ion Stoica Joseph E. Gonzalez RALM 42 127 0 12 Oct 2023
LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression Huiqiang Jiang Qianhui Wu Xufang Luo Dongsheng Li Chin-Yew Lin Yuqing Yang Lili Qiu RALM 121 192 0 10 Oct 2023
MemSum-DQA: Adapting An Efficient Long Document Extractive Summarizer for Document Question Answering Nianlong Gu Yingqiang Gao Richard H. R. Hahnloser RALM 46 0 0 10 Oct 2023
Scaling Laws of RoPE-based Extrapolation Xiaoran Liu Hang Yan Shuo Zhang Chen An Xipeng Qiu Dahua Lin 31 84 0 08 Oct 2023
Walking Down the Memory Maze: Beyond Context Limit through Interactive Reading Howard Chen Ramakanth Pasunuru Jason Weston Asli Celikyilmaz RALM 70 73 0 08 Oct 2023
LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extraction Litton J. Kurisinkel Nancy F. Chen 21 5 0 05 Oct 2023
Retrieval meets Long Context Large Language Models Peng Xu Ming-Yu Liu Xianchao Wu Lawrence C. McAfee Chen Zhu Zihan Liu Sandeep Subramanian Evelina Bakhturina M. Shoeybi Bryan Catanzaro RALM LRM 14 82 0 04 Oct 2023
Resolving Knowledge Conflicts in Large Language Models Yike Wang Shangbin Feng Heng Wang Weijia Shi Vidhisha Balachandran Tianxing He Yulia Tsvetkov 61 12 0 02 Oct 2023
GrowLength: Accelerating LLMs Pretraining by Progressively Growing Training Length Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Chia-Yuan Chang Xia Hu 35 11 0 01 Oct 2023