Linearity of Relation Decoding in Transformer Language Models

17 August 2023

Papers citing "Linearity of Relation Decoding in Transformer Language Models"

41 / 41 papers shown

Title
Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning Jingcheng Niu Subhabrata Dutta Ahmed Elshabrawy Harish Tayyar Madabushi Iryna Gurevych 110 1 0 16 May 2025
Do Large Language Models know who did what to whom? Joseph M. Denning Xiaohan Bryor Snefjella Idan A. Blank 198 1 0 23 Apr 2025
Exploring How LLMs Capture and Represent Domain-Specific Knowledge Mirian Hipolito Garcia Camille Couturier Daniel Madrigal Diaz Ankur Mallick Anastasios Kyrillidis Robert Sim Victor Rühle Saravan Rajmohan 62 1 0 23 Apr 2025
ConTrans: Weak-to-Strong Alignment Engineering via Concept Transplantation Weilong Dong Xinwei Wu Renren Jin Shaoyang Xu Deyi Xiong 103 9 0 31 Dec 2024
Towards Unifying Interpretability and Control: Evaluation via Intervention Usha Bhalla Suraj Srinivas Asma Ghandeharioun Himabindu Lakkaraju 88 11 0 07 Nov 2024
All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling Emanuele Marconato Sébastien Lachapelle Sebastian Weichwald Luigi Gresele 91 4 0 30 Oct 2024
Identifying Sub-networks in Neural Networks via Functionally Similar Representations Tian Gao Amit Dhurandhar Karthikeyan N. Ramamurthy Dennis L. Wei 73 0 0 21 Oct 2024
The Geometry of Numerical Reasoning: Language Models Compare Numeric Properties in Linear Subspaces Ahmed Oumar El-Shangiti Tatsuya Hiraoka Hilal AlQuabeh Benjamin Heinzerling Kentaro Inui 97 1 0 17 Oct 2024
Residual Stream Analysis with Multi-Layer SAEs Tim Lawson Lucy Farnik Conor Houghton Laurence Aitchison 72 5 0 06 Sep 2024
Relation Also Knows: Rethinking the Recall and Editing of Factual Associations in Auto-Regressive Transformer Language Models Xiyu Liu Zhengxiao Liu Naibin Gu Zheng Lin Wanli Ma Ji Xiang Weiping Wang KELM 68 0 0 27 Aug 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 142 32 0 02 Jul 2024
Knowledge Circuits in Pretrained Transformers Yunzhi Yao Ningyu Zhang Zekun Xi Meng Wang Ziwen Xu Shumin Deng Huajun Chen KELM 114 23 0 28 May 2024
Implicit In-context Learning Zhuowei Li Zihao Xu Ligong Han Yunhe Gao Song Wen Di Liu Hao Wang Dimitris N. Metaxas 98 3 0 23 May 2024
Language Models Implement Simple Word2Vec-style Vector Arithmetic Jack Merullo Carsten Eickhoff Ellie Pavlick KELM 59 65 0 25 May 2023
Dissecting Recall of Factual Associations in Auto-Regressive Language Models Mor Geva Jasmijn Bastings Katja Filippova Amir Globerson KELM 243 313 0 28 Apr 2023
Inspecting and Editing Knowledge Representations in Language Models Evan Hernandez Belinda Z. Li Jacob Andreas KELM 67 87 0 03 Apr 2023
Jump to Conclusions: Short-Cutting Transformers With Linear Transformations Alexander Yom Din Taelin Karidi Leshem Choshen Mor Geva 43 67 0 16 Mar 2023
Eliciting Latent Predictions from Transformers with the Tuned Lens Nora Belrose Zach Furman Logan Smith Danny Halawi Igor V. Ostrovsky Lev McKinney Stella Biderman Jacob Steinhardt 55 225 0 14 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.5K 13,182 0 27 Feb 2023
Crawling the Internal Knowledge-Base of Language Models Roi Cohen Mor Geva Jonathan Berant Amir Globerson 209 84 0 30 Jan 2023
Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models Peter Hase Joey Tianyi Zhou Been Kim Asma Ghandeharioun MILM 93 184 0 10 Jan 2023
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space Mor Geva Avi Caciularu Ke Wang Yoav Goldberg KELM 109 380 0 28 Mar 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 246 1,345 0 10 Feb 2022
Implicit Representations of Meaning in Neural Language Models Belinda Z. Li Maxwell Nye Jacob Andreas NAI MILM 57 176 0 01 Jun 2021
Knowledge Neurons in Pretrained Transformers Damai Dai Li Dong Y. Hao Zhifang Sui Baobao Chang Furu Wei KELM MU 81 451 0 18 Apr 2021
Probing Classifiers: Promises, Shortcomings, and Advances Yonatan Belinkov 266 443 0 24 Feb 2021
Transformer Feed-Forward Layers Are Key-Value Memories Mor Geva R. Schuster Jonathan Berant Omer Levy KELM 149 828 0 29 Dec 2020
Information-Theoretic Probing with Minimum Description Length Elena Voita Ivan Titov 73 275 0 27 Mar 2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts Colin Raffel Noam M. Shazeer KELM 106 890 0 10 Feb 2020
A Survey on Knowledge Graphs: Representation, Acquisition and Applications Shaoxiong Ji Shirui Pan Min Zhang Pekka Marttinen Philip S. Yu 298 1,989 0 02 Feb 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 493 42,407 0 03 Dec 2019
How Can We Know What Language Models Know? Zhengbao Jiang Frank F. Xu Jun Araki Graham Neubig KELM 130 1,405 0 28 Nov 2019
Designing and Interpreting Probes with Control Tasks John Hewitt Percy Liang 58 536 0 08 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 571 2,664 0 03 Sep 2019
Analysis Methods in Neural Language Processing: A Survey Yonatan Belinkov James R. Glass 77 557 0 21 Dec 2018
What you can cram into a single vector: Probing sentence embeddings for linguistic properties Alexis Conneau Germán Kruszewski Guillaume Lample Loïc Barrault Marco Baroni 331 893 0 03 May 2018
Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure Dieuwke Hupkes Sara Veldhoen Willem H. Zuidema 69 278 0 28 Nov 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 687 131,526 0 12 Jun 2017
What do Neural Machine Translation Models Learn about Morphology? Yonatan Belinkov Nadir Durrani Fahim Dalvi Hassan Sajjad James R. Glass 98 414 0 11 Apr 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 410 10,482 0 21 Jul 2016
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen G. Corrado J. Dean 3DV 668 31,489 0 16 Jan 2013