Attention Sorting Combats Recency Bias In Long Context Language Models

Attention Sorting Combats Recency Bias In Long Context Language Models

28 September 2023

A. Peysakhovich

Papers citing "Attention Sorting Combats Recency Bias In Long Context Language Models"

11 / 11 papers shown

Title
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation Chaofan Zhang Peng Hao Xiaoge Cao Xiaoshuai Hao Shaowei Cui Shuo Wang 32 0 0 14 May 2025
Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers Shijie Chen Bernal Jiménez Gutiérrez Yu Su 31 4 0 03 Oct 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 57 12 0 01 Jul 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 51 12 0 20 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 48 9 0 04 Jun 2024
Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding Zhenyu (Allen) Zhang Runjin Chen Shiwei Liu Zhewei Yao Olatunji Ruwase Beidi Chen Xiaoxia Wu Zhangyang Wang 34 26 0 05 Mar 2024
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku 64 2 0 16 Feb 2024
DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text Wenting Zhao Ye Liu Tong Niu Yao Wan Philip S. Yu Chenyu You Yingbo Zhou Semih Yavuz LRM 24 6 0 31 Oct 2023
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 398 8,559 0 28 Jan 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 698 0 27 Aug 2021
Distilling Knowledge from Reader to Retriever for Question Answering Gautier Izacard Edouard Grave RALM 185 251 0 08 Dec 2020