Attention Sorting Combats Recency Bias In Long Context Language Models

Attention Sorting Combats Recency Bias In Long Context Language Models

28 September 2023

A. Peysakhovich

Papers citing "Attention Sorting Combats Recency Bias In Long Context Language Models"

9 / 9 papers shown

Title
VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation Chaofan Zhang Peng Hao Xiaoge Cao Xiaoshuai Hao Shaowei Cui Shuo Wang 32 0 0 14 May 2025
Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers Shijie Chen Bernal Jiménez Gutiérrez Yu Su 31 4 0 03 Oct 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 57 12 0 01 Jul 2024
Insights into LLM Long-Context Failures: When Transformers Know but Don't Tell Taiming Lu Muhan Gao Kuai Yu Adam Byerly Daniel Khashabi 49 12 0 20 Jun 2024
Mitigate Position Bias in Large Language Models via Scaling a Single Dimension Yijiong Yu Huiqiang Jiang Xufang Luo Qianhui Wu Chin-Yew Lin Dongsheng Li Yuqing Yang Yongfeng Huang L. Qiu 46 9 0 04 Jun 2024
Where is the answer? Investigating Positional Bias in Language Model Knowledge Extraction Kuniaki Saito Kihyuk Sohn Chen-Yu Lee Yoshitaka Ushiku 62 2 0 16 Feb 2024
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 389 8,495 0 28 Jan 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 695 0 27 Aug 2021
Distilling Knowledge from Reader to Retriever for Question Answering Gautier Izacard Edouard Grave RALM 185 251 0 08 Dec 2020