Contextual Position Encoding: Learning to Count What's Important

Contextual Position Encoding: Learning to Count What's Important

29 May 2024

O. Yu. Golovneva

Jason Weston

Sainbayar Sukhbaatar

Papers citing "Contextual Position Encoding: Learning to Count What's Important"

9 / 9 papers shown

Title
A 2D Semantic-Aware Position Encoding for Vision Transformers Xi Chen Shiyang Zhou Muqi Huang Jiaxu Feng Yun Xiong ... Yuyao Zhang Huishuai Bao Sijia Peng Chong Li Feng Shi ViT 31 0 0 14 May 2025
TRA: Better Length Generalisation with Threshold Relative Attention Mattia Opper Roland Fernandez P. Smolensky Jianfeng Gao 46 0 0 29 Mar 2025
LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding Shen Zhang Yaning Tan Siyuan Liang Zhaowei Chen Linze Li ... Shuheng Li Zhenyu Zhao Caihua Chen Jiajun Liang Yao Tang 53 0 0 06 Mar 2025
Exposing Numeracy Gaps: A Benchmark to Evaluate Fundamental Numerical Abilities in Large Language Models Haoyang Li Xuejia Chen Zhanchao Xu Darian Li Nicole Hu ... Heng Chang Luyu Qiu C. Zhang Qing Li Lei Chen LRM ELM 45 1 0 16 Feb 2025
TULIP: Token-length Upgraded CLIP Ivona Najdenkoska Mohammad Mahdi Derakhshani Yuki M. Asano Nanne van Noord Marcel Worring Cees G. M. Snoek VLM 48 3 0 13 Oct 2024
KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches Jiayi Yuan Hongyi Liu Shaochen Zhong Yu-Neng Chuang ... Hongye Jin V. Chaudhary Zhaozhuo Xu Zirui Liu Xia Hu 46 18 0 01 Jul 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 57 12 0 01 Jul 2024
DeciMamba: Exploring the Length Extrapolation Potential of Mamba Assaf Ben-Kish Itamar Zimerman Shady Abu Hussein Nadav Cohen Amir Globerson Lior Wolf Raja Giryes Mamba 77 13 0 20 Jun 2024
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 698 0 27 Aug 2021