Long-context LLMs Struggle with Long In-context Learning

2 April 2024

Ge Zhang

Papers citing "Long-context LLMs Struggle with Long In-context Learning"

50 / 54 papers shown

Title
KG-HTC: Integrating Knowledge Graphs into LLMs for Effective Zero-shot Hierarchical Text Classification Qianbo Zang Christophe Zgrzendek Igor Tchappi Afshin Khadangi Johannes Sedlmeir VLM 35 0 0 08 May 2025
Recall with Reasoning: Chain-of-Thought Distillation for Mamba's Long-Context Memory and Extrapolation Junyu Ma Tianqing Fang Zhenru Zhang Hongming Zhang Haitao Mi Dong Yu ReLM RALM LRM 166 0 0 06 May 2025
Divide, Optimize, Merge: Fine-Grained LLM Agent Optimization at Scale Jiale Liu Yifan Zeng Shaokun Zhang Chi Zhang Malte Højmark-Bertelsen Marie Normann Gadeberg H. Wang Qingyun Wu 41 0 0 06 May 2025
Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models Matthew Dahl AILaw ELM 54 0 0 05 May 2025
Retrieval-augmented in-context learning for multimodal large language models in disease classification Zaifu Zhan Shuang Zhou Xiaoshan Zhou Yongkang Xiao Jun Wang Jiawen Deng He Zhu Yu Hou Rui Zhang 51 0 0 04 May 2025
Can LLMs Understand Time Series Anomalies? Zihao Zhou Rose Yu AI4TS 84 8 0 13 Mar 2025
HierarQ: Task-Aware Hierarchical Q-Former for Enhanced Video Understanding Shehreen Azad Vibhav Vineet Yogesh S Rawat VLM 163 1 0 11 Mar 2025
LongEval: A Comprehensive Analysis of Long-Text Generation Through a Plan-based Paradigm Siwei Wu Yong Li Xingwei Qu Rishi Ravikumar Yunshui Li Tyler Loakman Shanghaoran Quan Xiaoyong Wei Shanghaoran Quan Xiaoyong Wei R. Batista-Navarro Hongpeng Zhou 158 3 0 26 Feb 2025
Self-Memory Alignment: Mitigating Factual Hallucinations with Generalized Improvement Siyuan Zhang Y. Zhang Yinpeng Dong Hang Su HILM KELM 215 0 0 26 Feb 2025
LongAttn: Selecting Long-context Training Data via Token-level Attention Longyun Wu Dawei Zhu Guangxiang Zhao Zhuocheng Yu Junfeng Ran Xiangyu Wong Lin Sun Sujian Li 48 0 0 24 Feb 2025
From Perceptions to Decisions: Wildfire Evacuation Decision Prediction with Behavioral Theory-informed LLMs Ruxiao Chen Chenguang Wang Yuran Sun Xilei Zhao Susu Xu 95 1 0 24 Feb 2025
InductionBench: LLMs Fail in the Simplest Complexity Class Wenyue Hua Tyler Wong Sun Fei Liangming Pan Adam Jardine William Yang Wang LRM 73 3 0 20 Feb 2025
LCIRC: A Recurrent Compression Approach for Efficient Long-form Context and Query Dependent Modeling in LLMs Sumin An Junyoung Sung Wonpyo Park Chanjun Park Paul Hongsuck Seo 100 0 0 10 Feb 2025
ARISE: Iterative Rule Induction and Synthetic Data Generation for Text Classification Y. Meena Vaibhav Singh Ayush Maheshwari Amrith Krishna Ganesh Ramakrishnan AI4TS 121 0 0 09 Feb 2025
Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulation Songlin Xu Hao-Ning Wen Hongyi Pan Dallas Dominguez Dongyin Hu Xinyu Zhang 70 1 0 04 Feb 2025
Context-Aware Hierarchical Merging for Long Document Summarization Litu Ou Mirella Lapata MoMe 224 1 0 03 Feb 2025
CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification Cristiano Patrício Isabel Rio-Torto J. S. Cardoso Luís F. Teixeira João C. Neves VLM 242 1 0 21 Jan 2025
Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL Yeounoh Chung Gaurav Tarlok Kakkar Yu Gan Brenton Milne Fatma Ozcan RALM 63 5 0 21 Jan 2025
Revisiting In-Context Learning with Long Context Language Models Jinheon Baek Sun Jae Lee Prakhar Gupta Geunseob Oh Siddharth Dalmia 208 1 0 22 Dec 2024
AutoPrep: Natural Language Question-Aware Data Preparation with a Multi-Agent Framework Meihao Fan Ju Fan Nan Tang Lei Cao Guoliang Li Xiaoyong Du LMTD 123 0 0 10 Dec 2024
JPPO: Joint Power and Prompt Optimization for Accelerated Large Language Model Services Feiran You Hongyang Du Kaibin Huang Abbas Jamalipour 81 2 0 27 Nov 2024
What is Wrong with Perplexity for Long-context Language Modeling? Lizhe Fang Yifei Wang Zhaoyang Liu Chenheng Zhang Stefanie Jegelka Jinyang Gao Bolin Ding Yisen Wang 69 6 0 31 Oct 2024
From Single to Multi: How LLMs Hallucinate in Multi-Document Summarization Catarina G. Belem Pouya Pezeskhpour Hayate Iso Seiji Maekawa Nikita Bhutani Estevam R. Hruschka HILM 73 1 0 17 Oct 2024
In-Context Learning Enables Robot Action Prediction in LLMs Yida Yin Zekai Wang Yuvan Sharma Dantong Niu Trevor Darrell Roei Herzig LM&Ro 117 1 0 16 Oct 2024
Let's Ask GNN: Empowering Large Language Model for Graph In-Context Learning Zhengyu Hu Yichuan Li Zhengyu Chen Jiadong Wang Han Liu Kyumin Lee Kaize Ding GNN 205 1 0 09 Oct 2024
Narrative-of-Thought: Improving Temporal Reasoning of Large Language Models via Recounted Narratives Xinliang Frederick Zhang Nick Beauchamp Lu Wang LRM AI4CE 29 3 0 07 Oct 2024
In-context Learning in Presence of Spurious Correlations Hrayr Harutyunyan R. Darbinyan Samvel Karapetyan Hrant Khachatrian LRM 51 1 0 04 Oct 2024
How to Train Long-Context Language Models (Effectively) Tianyu Gao Alexander Wettig Howard Yen Danqi Chen RALM 72 38 0 03 Oct 2024
HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughly Howard Yen Tianyu Gao Minmin Hou Ke Ding Daniel Fleischer Peter Izsak Moshe Wasserblat Danqi Chen ALM ELM 62 25 0 03 Oct 2024
Bridging Context Gaps: Leveraging Coreference Resolution for Long Contextual Understanding Yanming Liu Xinyue Peng Jiannan Cao Shi Bo Yanxin Shen Tianyu Du Sheng Cheng Xun Wang Jianwei Yin Xuhong Zhang 68 9 0 02 Oct 2024
HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models Haoran Que Feiyu Duan Liqun He Yutao Mou Wangchunshu Zhou ... Ge Zhang Junran Peng Zhaoxiang Zhang Songyang Zhang Kai Chen LM&MA ELM VLM 51 11 0 24 Sep 2024
From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future Haolin Jin Linghan Huang Haipeng Cai Jun Yan Bo Li Huaming Chen 78 27 0 05 Aug 2024
Grammar-based Game Description Generation using Large Language Models Tsunehiko Tanaka Edgar Simo-Serra 62 2 0 24 Jul 2024
Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting Zilong Wang Zifeng Wang Long Le Huaixiu Steven Zheng Swaroop Mishra ... Anush Mattapalli Ankur Taly Jingbo Shang Chen-Yu Lee Tomas Pfister RALM 83 33 0 11 Jul 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 46 17 0 01 Jul 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 77 13 0 20 Jun 2024
BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack Yuri Kuratov Aydar Bulatov Petr Anokhin Ivan Rodkin Dmitry Sorokin Artyom Sorokin Andrey Kravchenko RALM ALM LRM ReLM ELM 51 59 0 14 Jun 2024
Chain of Agents: Large Language Models Collaborating on Long-Context Tasks Yusen Zhang Ruoxi Sun Yanfei Chen Tomas Pfister Rui Zhang Sercan Ö. Arik RALM AI4CE LLMAG 54 28 0 04 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 48 9 0 04 Jun 2024
Is In-Context Learning Sufficient for Instruction Following in LLMs? Hao Zhao Maksym Andriushchenko Francesco Croce Nicolas Flammarion 67 11 0 30 May 2024
(Perhaps) Beyond Human Translation: Harnessing Multi-Agent Collaboration for Translating Ultra-Long Literary Texts Minghao Wu Jiahao Xu Yulin Yuan Gholamreza Haffari Longyue Wang Weihua Luo Kaifu Zhang LLMAG 119 22 0 20 May 2024
Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers Rya Sanovar Srikant Bharadwaj Renée St. Amant Victor Rühle Saravan Rajmohan 61 6 0 17 May 2024
ADELIE: Aligning Large Language Models on Information Extraction Y. Qi Hao Peng Xiaozhi Wang Bin Xu Lei Hou Juanzi Li 41 7 0 08 May 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 131 434 0 13 Mar 2024
E^2-LLM: Efficient and Extreme Length Extension of Large Language Models Jiaheng Liu Zhiqi Bai Yuanxing Zhang Chenchen Zhang Yu Zhang ... Wenbo Su Tiezheng Ge Jie Fu Wenhu Chen Bo Zheng 48 8 0 13 Jan 2024
PoSE: Efficient Context Window Extension of LLMs via Positional Skip-wise Training Dawei Zhu Nan Yang Liang Wang Yifan Song Wenhao Wu Furu Wei Sujian Li 76 78 0 19 Sep 2023
On the Creativity of Large Language Models Giorgio Franceschelli Mirco Musolesi 72 52 0 27 Mar 2023
Resurrecting Recurrent Neural Networks for Long Sequences Antonio Orvieto Samuel L. Smith Albert Gu Anushan Fernando Çağlar Gülçehre Razvan Pascanu Soham De 88 268 0 11 Mar 2023
PoliGraph: Automated Privacy Policy Analysis using Knowledge Graphs (Journal Version) Hao Cui R. Trimananda A. Markopoulou Scott Jordan 51 17 0 13 Oct 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 253 1,073 0 05 Oct 2022