v1v2 (latest)

Finding Neurons in a Haystack: Case Studies with Sparse Probing

2 May 2023

Papers citing "Finding Neurons in a Haystack: Case Studies with Sparse Probing"

50 / 57 papers shown

Title
The Compositional Architecture of Regret in Large Language Models Xiangxiang Cui Shu Yang Tianjin Huang Wanyu Lin Lijie Hu Di Wang 22 0 0 18 Jun 2025
Distinct Computations Emerge From Compositional Curricula in In-Context Learning Jin Hwa Lee Andrew Kyle Lampinen Aaditya K. Singh Andrew Saxe 20 0 0 16 Jun 2025
Decomposing MLP Activations into Interpretable Features via Semi-Nonnegative Matrix Factorization Or Shafran Atticus Geiger Mor Geva MILM 93 0 0 12 Jun 2025
Transferring Features Across Language Models With Model Stitching Alan Chen Jack Merullo Alessandro Stolfo Ellie Pavlick 26 0 0 07 Jun 2025
Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safety Seongmin Lee Aeree Cho Grace C. Kim ShengYun Peng Mansi Phute Duen Horng Chau LM&MA AI4CE 63 0 0 05 Jun 2025
Precise In-Parameter Concept Erasure in Large Language Models Yoav Gur-Arieh Clara Suslik Yihuai Hong Fazl Barez Mor Geva KELM MU 86 0 0 28 May 2025
Pretrained LLMs Learn Multiple Types of Uncertainty Roi Cohen Omri Fahn Gerard de Melo 35 0 0 27 May 2025
Paths Not Taken: Understanding and Mending the Multilingual Factual Recall Pipeline Meng Lu Ruochen Zhang Carsten Eickhoff Ellie Pavlick HILM KELM LRM 72 0 0 26 May 2025
Understanding the learned look-ahead behavior of chess neural networks Diogo Cruz 15 0 0 26 May 2025
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Hao Chen Haoze Li Zhiqing Xiao Lirong Gao Qi Zhang Xiaomeng Hu Ningtao Wang Xing Fu Junbo Zhao 195 0 0 24 May 2025
Understanding Gated Neurons in Transformers from Their Input-Output Functionality Sebastian Gerstner Hinrich Schütze MILM FAtt 217 0 0 23 May 2025
Emergent Specialization: Rare Token Neurons in Language Models Jing Liu Haozheng Wang Yueheng Li MILM LRM 66 0 0 19 May 2025
Feature Hedging: Correlated Features Break Narrow Sparse Autoencoders David Chanin Tomáš Dulka Adrià Garriga-Alonso 56 0 0 16 May 2025
Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces Michael Pichat William Pogrund Paloma Pichat Judicael Poumay Armanouche Gasparian Samuel Demarchi Martin Corbet Alois Georgeon Michael Veillet-Guillem MILM 86 0 0 30 Apr 2025
I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? Yuhang Liu Dong Gong Erdun Gao Zhen Zhang Zhen Zhang Biwei Huang Anton van den Hengel Javen Qinfeng Shi Javen Qinfeng Shi 459 1 0 12 Mar 2025
SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability Adam Karvonen Can Rager Johnny Lin Curt Tigges Joseph Isaac Bloom ... Matthew Wearden Arthur Conmy Arthur Conmy Samuel Marks Neel Nanda MU 162 23 0 12 Mar 2025
Implicit Reasoning in Transformers is Reasoning through Shortcuts Tianhe Lin Jian Xie Siyu Yuan Deqing Yang ReLM LRM 153 3 0 10 Mar 2025
Exploiting Edited Large Language Models as General Scientific Optimizers Qitan Lv T. Liu Haoyu Wang 172 1 0 08 Mar 2025
Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective Yuko Nakagi Keigo Tada Sota Yoshino Shinji Nishimoto Yu Takagi LRM 128 0 0 28 Feb 2025
Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queries Tianyi Lorena Yan Robin Jia KELM MU 96 0 0 27 Feb 2025
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking Yifan Zhang Wenyu Du Dongming Jin Jie Fu Zhi Jin LRM 132 2 0 27 Feb 2025
Are Sparse Autoencoders Useful? A Case Study in Sparse Probing Subhash Kantamneni Joshua Engels Senthooran Rajamanoharan Max Tegmark Neel Nanda 141 17 0 23 Feb 2025
Soteria: Language-Specific Functional Parameter Steering for Multilingual Safety Alignment Somnath Banerjee Sayan Layek Pratyush Chatterjee Animesh Mukherjee Rima Hazra LLMSV 139 1 0 16 Feb 2025
The Representation and Recall of Interwoven Structured Knowledge in LLMs: A Geometric and Layered Analysis Ge Lei Samuel J. Cooper KELM 83 0 0 15 Feb 2025
The underlying structures of self-attention: symmetry, directionality, and emergent dynamics in Transformer training Matteo Saponati Pascal Sager Pau Vilimelis Aceituno Thilo Stadelmann Benjamin Grewe 7 1 0 15 Feb 2025
What is a Number, That a Large Language Model May Know It? Raja Marjieh Veniamin Veselovsky Thomas Griffiths Ilia Sucholutsky 453 3 0 03 Feb 2025
Discovering Chunks in Neural Embeddings for Interpretability Shuchen Wu Stephan Alaniz Eric Schulz Zeynep Akata 100 0 0 03 Feb 2025
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference Go Kamoda Benjamin Heinzerling Tatsuro Inaba Keito Kudo Keisuke Sakaguchi Kentaro Inui MILM 129 3 0 27 Jan 2025
Rethinking Evaluation of Sparse Autoencoders through the Representation of Polysemous Words Gouki Minegishi Hiroki Furuta Yusuke Iwasawa Y. Matsuo 103 3 0 09 Jan 2025
A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions Ola Shorinwa Zhiting Mei Justin Lidard Allen Z. Ren Anirudha Majumdar HILM LRM 123 19 0 07 Dec 2024
Improving Object Detection by Modifying Synthetic Data with Explainable AI Nitish Mital Simon Malzard Richard Walters Celso M. De Melo Raghuveer Rao Victoria Nockles 164 0 0 02 Dec 2024
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering Zeping Yu Sophia Ananiadou 479 2 0 17 Nov 2024
Math Neurosurgery: Isolating Language Models' Math Reasoning Abilities Using Only Forward Passes Bryan R Christ Zack Gottesman Jonathan Kropko Thomas Hartvigsen LRM 136 4 0 22 Oct 2024
On the Role of Attention Heads in Large Language Model Safety Zhenhong Zhou Haiyang Yu Xinghua Zhang Rongwu Xu Fei Huang Kun Wang Yang Liu Sihang Li Yongbin Li 156 10 0 17 Oct 2024
Mechanistic Permutability: Match Features Across Layers Nikita Balagansky Ian Maksimov Daniil Gavrilov 82 7 0 10 Oct 2024
From Tokens to Words: On the Inner Lexicon of LLMs Guy Kaplan Matanel Oren Yuval Reif Roy Schwartz 105 14 0 08 Oct 2024
A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders David Chanin James Wilken-Smith Tomáš Dulka Hardik Bhatnagar Joseph Bloom Joseph Isaac Bloom 128 37 0 22 Sep 2024
From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning Wei Chen Zhen Huang Liang Xie Binbin Lin Houqiang Li ... Deng Cai Yonggang Zhang Wenxiao Wang Xu Shen Jieping Ye 145 10 0 03 Sep 2024
Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neuroscience Zhonghao He Jascha Achterberg Katie Collins Kevin K. Nejad Danyal Akarca ... Chole Li Kai J. Sandbrink Stephen Casper Anna Ivanova Grace W. Lindsay AI4CE 97 2 0 22 Aug 2024
Knowledge in Superposition: Unveiling the Failures of Lifelong Knowledge Editing for Large Language Models Chenhui Hu Pengfei Cao Yubo Chen Kang Liu Jun Zhao KELM 129 3 0 14 Aug 2024
Validating Mechanistic Interpretations: An Axiomatic Approach Nils Palumbo Ravi Mangal Zifan Wang Saranya Vijayakumar Corina S. Pasareanu Somesh Jha 106 1 0 18 Jul 2024
A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models Daking Rai Yilun Zhou Shi Feng Abulhair Saparov Ziyu Yao 187 33 0 02 Jul 2024
The Remarkable Robustness of LLMs: Stages of Inference? Vedang Lad Wes Gurnee Max Tegmark Max Tegmark 113 53 0 27 Jun 2024
Talking Heads: Understanding Inter-layer Communication in Transformer Language Models Jack Merullo Carsten Eickhoff Ellie Pavlick 146 16 0 13 Jun 2024
The Geometry of Categorical and Hierarchical Concepts in Large Language Models Kiho Park Yo Joong Choe Yibo Jiang Victor Veitch 131 41 0 03 Jun 2024
How Do Transformers "Do" Physics? Investigating the Simple Harmonic Oscillator Subhash Kantamneni Ziming Liu Max Tegmark 170 2 0 23 May 2024
Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questions Ruizhe Li Yanjun Gao KELM 80 8 0 06 May 2024
A Multimodal Automated Interpretability Agent Tamar Rott Shaham Sarah Schwettmann Franklin Wang Achyuta Rajaram Evan Hernandez Jacob Andreas Antonio Torralba 219 27 0 22 Apr 2024
Towards Uncovering How Large Language Model Works: An Explainability Perspective Haiyan Zhao Fan Yang Bo Shen Himabindu Lakkaraju Jundong Li 91 13 0 16 Feb 2024
Successor Heads: Recurring, Interpretable Attention Heads In The Wild Rhys Gould Euan Ong George Ogden Arthur Conmy LRM 39 52 0 14 Dec 2023