CLEX: Continuous Length Extrapolation for Large Language Models

25 October 2023

Xin Li

Papers citing "CLEX: Continuous Length Extrapolation for Large Language Models"

24 / 24 papers shown

Title
Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation Yi Lu Wanxu Zhao Xin Zhou Chenxin An Cong Wang ... Jun Zhao Tao Ji Tao Gui Qi Zhang Xuanjing Huang 41 0 0 26 Apr 2025
The Role of Sparsity for Length Generalization in Transformers Noah Golowich Samy Jelassi David Brandfonbrener Sham Kakade Eran Malach 42 0 0 24 Feb 2025
WildLong: Synthesizing Realistic Long-Context Instruction Data at Scale Jiaxi Li Xingxing Zhang Xun Wang Xiaolong Huang Li Dong Liang Wang Si-Qing Chen Wei Lu Furu Wei SyDa 201 0 0 23 Feb 2025
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training Haonan Wang Qian Liu Chao Du Tongyao Zhu Cunxiao Du Kenji Kawaguchi Tianyu Pang 115 6 0 20 Nov 2024
TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network Nouf Alabbasi Omar Erak Omar Alhussein Ismail Lotfi Sami Muhaidat Merouane Debbah RALM 189 0 0 04 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 69 6 0 31 Oct 2024
DAPE V2: Process Attention Score as Feature Map for Length Extrapolation Chuanyang Zheng Yihang Gao Han Shi Jing Xiong Jiankai Sun ... Xiaozhe Ren Michael Ng Xin Jiang Zhenguo Li Yu Li 36 2 0 07 Oct 2024
A Controlled Study on Long Context Extension and Generalization in LLMs Yi Lu Jing Nathan Yan Songlin Yang Justin T. Chiu Siyu Ren Fei Yuan Wenting Zhao Zhiyong Wu Alexander M. Rush 35 9 0 18 Sep 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 31 18 0 12 Jul 2024
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models Jingyao Li Han Shi Xin Jiang Zhenguo Li Hong Xu Jiaya Jia LRM 35 2 0 11 Jun 2024
MambaLRP: Explaining Selective State Space Sequence Models F. Jafari G. Montavon Klaus-Robert Müller Oliver Eberle Mamba 62 9 0 11 Jun 2024
LongSSM: On the Length Extension of State-space Models in Language Modelling Shida Wang 40 0 0 04 Jun 2024
Quest: Query-centric Data Synthesis Approach for Long-context Scaling of Large Language Model Chaochen Gao Xing Wu Qingfang Fu Songlin Hu SyDa 34 5 0 30 May 2024
Can Perplexity Reflect Large Language Model's Ability in Long Text Understanding? Yutong Hu Quzhe Huang Mingxu Tao Chen Zhang Yansong Feng 34 23 0 09 May 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu (Allen) Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 34 26 0 05 Mar 2024
Training-Free Long-Context Scaling of Large Language Models Chen An Fei Huang Jun Zhang Shansan Gong Xipeng Qiu Chang Zhou Lingpeng Kong ALM LRM 40 35 0 27 Feb 2024
InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memory Chaojun Xiao Pengle Zhang Xu Han Guangxuan Xiao Yankai Lin Zhengyan Zhang Zhiyuan Liu Maosong Sun LLMAG 47 35 0 07 Feb 2024
Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models Xindi Wang Mahsa Salmani Parsa Omidi Xiangyu Ren Mehdi Rezagholizadeh A. Eshaghi LRM 34 35 0 03 Feb 2024
Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation Zhenyu He Guhao Feng Shengjie Luo Kai-Bo Yang Liwei Wang Jingjing Xu Zhi Zhang Hongxia Yang Di He 32 14 0 29 Jan 2024
With Greater Text Comes Greater Necessity: Inference-Time Training Helps Long Text Generation Y. Wang D. Ma D. Cai RALM 49 18 0 21 Jan 2024
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning Hongye Jin Xiaotian Han Jingfeng Yang Zhimeng Jiang Zirui Liu Chia-Yuan Chang Huiyuan Chen Xia Hu 42 101 0 02 Jan 2024
Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey Yunpeng Huang Jingwei Xu Junyu Lai Zixu Jiang Taolue Chen ... Xiaoxing Ma Lijuan Yang Zhou Xin Shupeng Li Penghao Zhao LLMAG KELM 41 55 0 21 Nov 2023
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 698 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 282 1,996 0 31 Dec 2020