Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models

28 March 2024

Rui Yan

Papers citing "Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models"

26 / 26 papers shown

Title
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection Kai Hua Steven Wu Ge Zhang Ke Shen LRM 28 0 0 12 May 2025
Platonic Grounding for Efficient Multimodal Language Models Moulik Choraria Xinbo Wu Akhil Bhimaraju Nitesh Sekhar Yue Wu Xu Zhang Prateek Singhal L. Varshney 59 0 0 27 Apr 2025
Taming Knowledge Conflicts in Language Models Gaotang Li Yuzhong Chen Hanghang Tong KELM 49 1 0 14 Mar 2025
Revealing and Mitigating Over-Attention in Knowledge Editing Pinzheng Wang Zecheng Tang Keyan Zhou J. Li Qiaoming Zhu M. Zhang KELM 120 2 0 21 Feb 2025
Hymba: A Hybrid-head Architecture for Small Language Models Xin Dong Y. Fu Shizhe Diao Wonmin Byeon Zijia Chen ... Min-Hung Chen Yoshi Suhara Y. Lin Jan Kautz Pavlo Molchanov Mamba 100 21 0 20 Nov 2024
Information Anxiety in Large Language Models Prasoon Bajpai Sarah Masud Tanmoy Chakraborty 37 0 0 16 Nov 2024
ResiDual Transformer Alignment with Spectral Decomposition Lorenzo Basile Valentino Maiorca Luca Bortolussi Emanuele Rodolà Francesco Locatello 48 1 0 31 Oct 2024
LargePiG: Your Large Language Model is Secretly a Pointer Generator ZhongXiang Sun Zihua Si Xiaoxue Zang Kai Zheng Yang Song Xiao Zhang Jun Xu HILM RALM 42 0 0 15 Oct 2024
PEAR: Position-Embedding-Agnostic Attention Re-weighting Enhances Retrieval-Augmented Generation with Zero Inference Overhead Tao Tan Yining Qian Ang Lv Hongzhan Lin Songhao Wu Yongbo Wang Feng Wang Jingtong Wu Xin Lu Rui Yan 22 1 0 29 Sep 2024
Attention Heads of Large Language Models: A Survey Zifan Zheng Yezhaohui Wang Yuxin Huang Shichao Song Mingchuan Yang Bo Tang Feiyu Xiong Zhiyu Li LRM 58 22 0 05 Sep 2024
Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules Zhuocheng Gong Ang Lv Jian-Yu Guan Junxi Yan Wei Yu Wu Huishuai Zhang Minlie Huang Dongyan Zhao Rui Yan MoE 52 6 0 09 Jul 2024
$$\text{Memory}^3$: Language Modeling with Explicit Memory$ $\text{Memory}^3$ : Language Modeling with Explicit Memory Hongkang Yang Zehao Lin Wenjin Wang Hao Wu Zhiyu Li ... Yu Yu Kai Chen Feiyu Xiong Linpeng Tang Weinan E 50 11 0 01 Jul 2024
Mixture of In-Context Experts Enhance LLMs' Long Context Awareness Hongzhan Lin Ang Lv Yuhan Chen Chen Zhu Yang Song Hengshu Zhu Rui Yan 29 9 0 28 Jun 2024
Knowledge Circuits in Pretrained Transformers Yunzhi Yao Ningyu Zhang Zekun Xi Meng Wang Ziwen Xu Shumin Deng Huajun Chen KELM 64 20 0 28 May 2024
Monotonic Representation of Numeric Properties in Language Models Benjamin Heinzerling Kentaro Inui KELM MILM 45 9 0 15 Mar 2024
Do Llamas Work in English? On the Latent Language of Multilingual Transformers Chris Wendler V. Veselovsky Giovanni Monea Robert West 56 97 0 16 Feb 2024
CharacterEval: A Chinese Benchmark for Role-Playing Conversational Agent Evaluation Quan Tu Shilong Fan Zihang Tian Rui Yan 81 58 0 02 Jan 2024
Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use Yuhan Chen Ang Lv Ting-En Lin C. Chen Yuchuan Wu Fei Huang Yongbin Li Rui Yan 21 24 0 07 Dec 2023
Large Knowledge Model: Perspectives and Challenges Huajun Chen KELM 53 9 0 05 Dec 2023
Characterizing Mechanisms for Factual Recall in Language Models Qinan Yu Jack Merullo Ellie Pavlick KELM 42 23 0 24 Oct 2023
An Adversarial Example for Direct Logit Attribution: Memory Management in gelu-4l James Dao Yeu-Tong Lau Can Rager Jett Janiak 35 5 0 11 Oct 2023
Dissecting Recall of Factual Associations in Auto-Regressive Language Models Mor Geva Jasmijn Bastings Katja Filippova Amir Globerson KELM 191 261 0 28 Apr 2023
Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small Kevin Wang Alexandre Variengien Arthur Conmy Buck Shlegeris Jacob Steinhardt 212 496 0 01 Nov 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 250 460 0 24 Sep 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 125 318 0 21 Sep 2022
Fine-Tuning Language Models from Human Preferences Daniel M. Ziegler Nisan Stiennon Jeff Wu Tom B. Brown Alec Radford Dario Amodei Paul Christiano G. Irving ALM 280 1,595 0 18 Sep 2019