Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell

20 June 2024

Papers citing "Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell"

25 / 25 papers shown

Title
Lost in the Haystack: Smaller Needles are More Difficult for LLMs to Find Owen Bianchi Mathew J. Koretsky Maya Willey Chelsea X. Alvarado Tanay Nayak Adi Asija Nicole Kuznetsov M. Nalls F. Faghri Daniel Khashabi 67 0 0 23 May 2025
Savaal: Scalable Concept-Driven Question Generation to Enhance Human Learning Kimia Noorbakhsh Joseph Chandler Pantea Karimi M. Alizadeh H. Balakrishnan LRM 92 1 0 18 Feb 2025
A Deep Dive Into Large Language Model Code Generation Mistakes: What and Why? QiHong Chen Jiawei Li Jiecheng Deng Jiachen Yu Justin Tian Jin Chen Iftekhar Ahmed 120 1 0 03 Nov 2024
MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation Chenxi Wang Xiang Chen N. Zhang Bozhong Tian Haoming Xu Shumin Deng Ningyu Zhang MLLM LRM 205 10 0 15 Oct 2024
Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers Yibo Jiang Goutham Rajendran Pradeep Ravikumar Bryon Aragam CLL KELM 76 8 0 26 Jun 2024
Exploring Concept Depth: How Large Language Models Acquire Knowledge at Different Layers? Mingyu Jin Qinkai Yu Jingyuan Huang Qingcheng Zeng Zhenting Wang ... Yanda Meng Kaize Ding Fan Yang Jundong Li Yongfeng Zhang 65 21 0 10 Apr 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 83 35 0 05 Mar 2024
How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Study Tianjie Ju Weiwei Sun Wei Du Xinwei Yuan Zhaochun Ren Gongshen Liu KELM 59 31 0 25 Feb 2024
LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Yiran Ding Li Lyna Zhang Chengruidong Zhang Yuanyuan Xu Ning Shang Jiahang Xu Fan Yang Mao Yang RALM 86 156 0 21 Feb 2024
$$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens$ $\infty$ Bench: Extending Long Context Evaluation Beyond 100K Tokens Xinrong Zhang Yingfa Chen Shengding Hu Zihang Xu Junhao Chen ... Xu Han Zhen Leng Thai Shuo Wang Zhiyuan Liu Maosong Sun RALM LRM 94 188 0 21 Feb 2024
Do Androids Know They're Only Dreaming of Electric Sheep? Sky CH-Wang Benjamin Van Durme Jason Eisner Chris Kedzie HILM 58 35 0 28 Dec 2023
Cognitive Dissonance: Why Do Language Model Outputs Disagree with Internal Representations of Truthfulness? Kevin Liu Stephen Casper Dylan Hadfield-Menell Jacob Andreas HILM 93 38 0 27 Nov 2023
Primacy Effect of ChatGPT Yiwei Wang Yujun Cai Muhao Chen Yuxuan Liang Bryan Hooi ALM AI4MH LRM 58 17 0 20 Oct 2023
The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets Samuel Marks Max Tegmark HILM 140 216 0 10 Oct 2023
Attention Sorting Combats Recency Bias In Long Context Language Models A. Peysakhovich Adam Lerer LRM RALM 91 52 0 28 Sep 2023
Lost in the Middle: How Language Models Use Long Contexts Nelson F. Liu Kevin Lin John Hewitt Ashwin Paranjape Michele Bevilacqua Fabio Petroni Percy Liang RALM 106 1,600 0 06 Jul 2023
Extending Context Window of Large Language Models via Positional Interpolation Shouyuan Chen Sherman Wong Liangjian Chen Yuandong Tian 151 532 0 27 Jun 2023
Inference-Time Intervention: Eliciting Truthful Answers from a Language Model Kenneth Li Oam Patel Fernanda Viégas Hanspeter Pfister Martin Wattenberg KELM HILM 90 561 0 06 Jun 2023
ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding Uri Shaham Maor Ivgi Avia Efrat Jonathan Berant Omer Levy VLM 76 138 0 23 May 2023
The Internal State of an LLM Knows When It's Lying A. Azaria Tom Michael Mitchell HILM 267 340 0 26 Apr 2023
SCROLLS: Standardized CompaRison Over Long Language Sequences Uri Shaham Elad Segal Maor Ivgi Avia Efrat Ori Yoran ... Ankit Gupta Wenhan Xiong Mor Geva Jonathan Berant Omer Levy RALM 84 137 0 10 Jan 2022
UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction Leland McInnes John Healy James Melville 160 9,432 0 09 Feb 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 701 131,652 0 12 Jun 2017
Understanding intermediate layers using linear classifier probes Guillaume Alain Yoshua Bengio FAtt 158 949 0 05 Oct 2016
A Tutorial on Principal Component Analysis Jonathon Shlens 87 2,496 0 03 Apr 2014