Towards Universality: Studying Mechanistic Similarity Across Language
Model Architectures

v1v2 (latest)

Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures

9 October 2024

Junxuan Wang

Xipeng Qiu

ArXiv (abs)PDF HTML

Papers citing "Towards Universality: Studying Mechanistic Similarity Across Language Model Architectures"

15 / 15 papers shown

Title
Layers at Similar Depths Generate Similar Activations Across LLM Architectures Christopher Wolfram Aaron Schein 92 2 0 03 Apr 2025
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 158 33 0 02 Jul 2024
Interpreting the Second-Order Effects of Neurons in CLIP Yossi Gandelsman Alexei A. Efros Jacob Steinhardt MILM 114 24 0 06 Jun 2024
Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality Tri Dao Albert Gu Mamba 108 513 0 31 May 2024
Function Vectors in Large Language Models Eric Todd Millicent Li Arnab Sen Sharma Aaron Mueller Byron C. Wallace David Bau 55 116 0 23 Oct 2023
RWKV: Reinventing RNNs for the Transformer Era Bo Peng Eric Alcaide Quentin G. Anthony Alon Albalak Samuel Arcadinho ... Qihang Zhao P. Zhou Qinghua Zhou Jian Zhu Rui-Jie Zhu 227 597 0 22 May 2023
Similarity of Neural Network Models: A Survey of Functional and Representational Measures Max Klabunde Tobias Schumacher M. Strohmaier Florian Lemmerich 139 73 0 10 May 2023
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 316 524 0 24 Sep 2022
Toy Models of Superposition Nelson Elhage Tristan Hume Catherine Olsson Nicholas Schiefer T. Henighan ... Sam McCandlish Jared Kaplan Dario Amodei Martin Wattenberg C. Olah AAML MILM 183 377 0 21 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 880 13,148 0 04 Mar 2022
Transformer Quality in Linear Time Weizhe Hua Zihang Dai Hanxiao Liu Quoc V. Le 124 229 0 21 Feb 2022
Locating and Editing Factual Associations in GPT Kevin Meng David Bau A. Andonian Yonatan Belinkov KELM 248 1,357 0 10 Feb 2022
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 452 2,113 0 31 Dec 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 826 42,332 0 28 May 2020
Similarity of Neural Network Representations Revisited Simon Kornblith Mohammad Norouzi Honglak Lee Geoffrey E. Hinton 141 1,429 0 01 May 2019