Title
When Precision Meets Position: BFloat16 Breaks Down RoPE in Long-Context Training Haonan Wang Qian Liu Chao Du Tongyao Zhu Cunxiao Du Kenji Kawaguchi Tianyu Pang 115 6 0 20 Nov 2024
LIFBench: Evaluating the Instruction Following Performance and Stability of Large Language Models in Long-Context Scenarios Xiaodong Wu Minhao Wang Yichen Liu Xiaoming Shi He Yan Xiangju Lu Junmin Zhu Wei Zhang 207 3 0 11 Nov 2024
Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks? Jonathan Roberts Kai Han Samuel Albanie LLMAG 192 0 0 07 Nov 2024
TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection Wei Wu Zhuoshi Pan Chao Wang L. Chen Y. Bai Kun Fu Zehua Wang Hui Xiong Hui Xiong LLMAG 39 5 0 05 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 69 6 0 31 Oct 2024
ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference Hanshi Sun Li-Wen Chang Yiyuan Ma Wenlei Bao Ningxin Zheng Xin Liu Harry Dong Yuejie Chi Beidi Chen VLM 88 16 0 28 Oct 2024
Two are better than one: Context window extension with multi-grained self-injection Wei Han Pan Zhou Soujanya Poria Shuicheng Yan 29 0 0 25 Oct 2024
ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage Taewhoo Lee Chanwoong Yoon Kyochul Jang Donghyeon Lee Minju Song Hyunjae Kim Jaewoo Kang ELM 35 1 0 22 Oct 2024
MoDification: Mixture of Depths Made Easy C. Zhang M. Zhong Qimeng Wang Xuantao Lu Zheyu Ye ... Yan Gao Yao Hu Kehai Chen Min Zhang Dawei Song VLM MoE 38 2 0 18 Oct 2024
An Evolved Universal Transformer Memory Edoardo Cetin Qi Sun Tianyu Zhao Yujin Tang 188 0 0 17 Oct 2024
Holistic Reasoning with Long-Context LMs: A Benchmark for Database Operations on Massive Textual Data Seiji Maekawa Hayate Iso Nikita Bhutani RALM 110 1 0 15 Oct 2024
$LLM$\times$MapReduce: Simplified Long-Sequence Processing using Large Language Models$ LLM $\times$ MapReduce: Simplified Long-Sequence Processing using Large Language Models Zihan Zhou Chong Li Xinyi Chen Shuo Wang Yu Chao ... Zhixing Tan Xu Han Xiaodong Shi Zhiyuan Liu Maosong Sun 26 0 0 12 Oct 2024
Stuffed Mamba: State Collapse and State Capacity of RNN-Based Long-Context Modeling Yingfa Chen Xinrong Zhang Shengding Hu Xu Han Zhiyuan Liu Maosong Sun Mamba 59 2 0 09 Oct 2024
MLissard: Multilingual Long and Simple Sequential Reasoning Benchmarks M. Bueno R. Lotufo Rodrigo Nogueira LRM 31 0 0 08 Oct 2024
Forgetting Curve: A Reliable Method for Evaluating Memorization Capability for Long-context Models Xinyu Liu Runsong Zhao Pengcheng Huang Chunyang Xiao Bei Li Jingang Wang Tong Xiao Jingbo Zhu 30 0 0 07 Oct 2024
MathHay: An Automated Benchmark for Long-Context Mathematical Reasoning in LLMs Lei Wang Shan Dong Yuhui Xu Hanze Dong Yalu Wang Amrita Saha Ee-Peng Lim Caiming Xiong Doyen Sahoo LRM 48 2 0 07 Oct 2024
LongGenBench: Long-context Generation Benchmark Xiang Liu Peijie Dong Xuming Hu Xiaowen Chu RALM 55 8 0 05 Oct 2024
ALR $^2$ : A Retrieve-then-Reason Framework for Long-context Question Answering Huayang Li Pat Verga Priyanka Sen Bowen Yang Vijay Viswanathan Patrick Lewis Taro Watanabe Yixuan Su RALM LRM 46 8 0 04 Oct 2024
L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding? Zecheng Tang Keyan Zhou Juntao Li Baibei Ji Jianye Hou Min Zhang 47 2 0 03 Oct 2024
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Howard Yen Tianyu Gao Minmin Hou Ke Ding Daniel Fleischer Peter Izsak Moshe Wasserblat Danqi Chen ALM ELM 62 25 0 03 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 72 38 0 03 Oct 2024
Locret: Enhancing Eviction in Long-Context LLM Inference with Trained Retaining Heads on Consumer-Grade Devices Yuxiang Huang Binhang Yuan Xu Han Chaojun Xiao Zhiyuan Liu RALM 87 1 0 02 Oct 2024
Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding Yanming Liu Xinyue Peng Jiannan Cao Shi Bo Yanxin Shen Tianyu Du Sheng Cheng Xun Wang Jianwei Yin Xuhong Zhang 71 9 0 02 Oct 2024
Beyond Prompts: Dynamic Conversational Benchmarking of Large Language Models David Castillo-Bolado Joseph Davidson Finlay Gray Marek Rosa 34 3 0 30 Sep 2024
2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models Jia-Nan Li Jian Guan Wei Wu Zhengtao Yu Rui Yan LMTD 44 1 0 29 Sep 2024
Towards LifeSpan Cognitive Systems Yu Wang Chi Han Tongtong Wu Xiaoxin He Wangchunshu Zhou ... Zexue He Wei Wang Gholamreza Haffari Heng Ji Julian McAuley KELM CLL 188 1 0 20 Sep 2024
Retrieval Or Holistic Understanding? Dolce: Differentiate Our Long Context Evaluation Tasks Zi Yang 33 0 0 10 Sep 2024
LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs Yuhao Wu Ming Shan Hee Zhiqing Hu Roy Ka-Wei Lee RALM 35 8 0 03 Sep 2024
In Defense of RAG in the Era of Long-Context Language Models Tan Yu Anbang Xu Rama Akkiraju RALM 3DV 29 24 0 03 Sep 2024
FocusLLM: Scaling LLM's Context by Parallel Decoding Zhenyu Li Yike Zhang Tengyu Pan Yutao Sun Zhichao Duan Junjie Fang Rong Han Zixuan Wang Jianyong Wang 31 2 0 21 Aug 2024
Multilingual Needle in a Haystack: Investigating Long-Context Behavior of Multilingual Large Language Models Amey Hengle Prasoon Bajpai Soham Dan Tanmoy Chakraborty LRM 37 2 0 19 Aug 2024
Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption Shi Luohe Hongyi Zhang Yao Yao Z. Li Zhao Hai 31 34 0 25 Jul 2024
Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach Zhuowan Li Cheng-rong Li Mingyang Zhang Qiaozhu Mei Michael Bendersky 3DV RALM 60 36 0 23 Jul 2024
On the Design and Analysis of LLM-Based Algorithms Yanxi Chen Yaliang Li Bolin Ding Jingren Zhou 51 5 0 20 Jul 2024
ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities Peng Xu Ming-Yu Liu Xianchao Wu Zihan Liu M. Shoeybi Mohammad Shoeybi Bryan Catanzaro RALM 52 14 0 19 Jul 2024
Human-like Episodic Memory for Infinite Context LLMs Z. Fountas Martin A Benfeghoul Adnan Oomerjee Fenia Christopoulou Gerasimos Lampouras Haitham Bou-Ammar Jun Wang 31 18 0 12 Jul 2024
Evaluating Language Model Context Windows: A "Working Memory" Test and Inference-time Correction Amanda Dsouza Christopher Glaze Changho Shin Frederic Sala LRM 24 0 0 04 Jul 2024
MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention Huiqiang Jiang Yucheng Li Chengruidong Zhang Qianhui Wu Xufang Luo ... Amir H. Abdi Dongsheng Li Chin-Yew Lin Yuqing Yang L. Qiu 72 84 0 02 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 46 18 0 01 Jul 2024
MMLongBench-Doc: Benchmarking Long-context Document Understanding with Visualizations Yubo Ma Yuhang Zang Liangyu Chen Meiqi Chen Yizhu Jiao ... Liangming Pan Yu-Gang Jiang Jiaqi Wang Yixin Cao Aixin Sun ELM RALM VLM 39 25 0 01 Jul 2024
Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP Omer Goldman Alon Jacovi Aviv Slobodkin Aviya Maimon Ido Dagan Reut Tsarfaty 66 11 0 29 Jun 2024
Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMs Lei Zhang Yunshui Li Jiaming Li Xiaobo Xia Jiaxi Yang Run Luo Minzheng Wang Longze Chen Junhao Liu Min Yang 40 1 0 26 Jun 2024
Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA Minzheng Wang Longze Chen Cheng Fu Shengyi Liao Xinghua Zhang ... Run Luo Yunshui Li Min Yang Fei Huang Yongbin Li RALM 54 44 0 25 Jun 2024
One Thousand and One Pairs: A "novel" challenge for long-context language models Marzena Karpinska Katherine Thai Kyle Lo Tanya Goyal Mohit Iyyer LRM 43 41 0 24 Jun 2024
MedOdyssey: A Medical Domain Benchmark for Long Context Evaluation Up to 200K Tokens Yongqi Fan Hongli Sun Kui Xue Xiaofan Zhang Shaoting Zhang Tong Ruan 47 0 0 21 Jun 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 51 12 0 20 Jun 2024
What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling Yutong Hu Quzhe Huang Kangcheng Luo Yansong Feng 56 1 0 17 Jun 2024
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack Yuri Kuratov Aydar Bulatov Petr Anokhin Ivan Rodkin Dmitry Sorokin Artyom Sorokin Andrey Kravchenko RALM ALM LRM ReLM ELM 51 61 0 14 Jun 2024
QuickLLaMA: Query-aware Inference Acceleration for Large Language Models Jingyao Li Han Shi Xin Jiang Zhenguo Li Hong Xu Jiaya Jia LRM 35 2 0 11 Jun 2024
RepoQA: Evaluating Long Context Code Understanding Jiawei Liu Jia Le Tian Vijay Daita Yuxiang Wei Yifeng Ding Yuhan Katherine Wang Jun Yang Lingming Zhang LLMAG 39 17 0 10 Jun 2024