What Does BERT Look At? An Analysis of BERT's Attention

11 June 2019

Kevin Clark

Urvashi Khandelwal

Omer Levy

Christopher D. Manning

MILM

ArXiv (abs)PDF HTML

Papers citing "What Does BERT Look At? An Analysis of BERT's Attention"

50 / 906 papers shown

Title
Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations Ananth Agarwal Jasper Jian Christopher D. Manning Shikhar Murty 5 0 0 20 Jun 2025
Knee-Deep in C-RASP: A Transformer Depth Hierarchy Andy Yang Michaël Cadilhac David Chiang 5 0 0 19 Jun 2025
Under the Shadow of Babel: How Language Shapes Reasoning in LLMs Chenxi Wang Y. Zhang Lang Gao Zixiang Xu Zirui Song Yanbo Wang Xiuying Chen 5 0 0 19 Jun 2025
A correlation-permutation approach for speech-music encoders model merging Fabian Ritter-Gutierrez Yi-Cheng Lin Jeremy H.M Wong Hung-yi Lee Eng Siong Chng Nancy F. Chen MoMe 56 0 0 13 Jun 2025
RollingQ: Reviving the Cooperation Dynamics in Multimodal Transformer Haotian Ni Yake Wei Hang Liu Gong Chen Chong Peng Hao Lin Di Hu OffRL 56 0 0 13 Jun 2025
Mechanistic Decomposition of Sentence Representations Matthieu Tehenan Vikram Natarajan Jonathan Michala Milton Lin Juri Opitz 21 0 0 04 Jun 2025
It's Not a Walk in the Park! Challenges of Idiom Translation in Speech-to-text Systems Iuliia Zaitova Badr M. Abdullah Wei Xue Dietrich Klakow Bernd Möbius T. Avgustinova 54 0 0 03 Jun 2025
Revisiting LRP: Positional Attribution as the Missing Ingredient for Transformer Explainability Yarden Bakish Itamar Zimerman Hila Chefer Lior Wolf 9 0 0 02 Jun 2025
Explainable Depression Detection using Masked Hard Instance Mining Patawee Prakrankamanant Shinji Watanabe Ekapol Chuangsuwanich 27 0 0 30 May 2025
Transformers Are Universally Consistent Sagar Ghosh Kushal Bose Swagatam Das 28 0 0 30 May 2025
Limited Generalizability in Argument Mining: State-Of-The-Art Models Learn Datasets, Not Arguments Marc Feger Katarina Boland Stefan Dietze 23 0 0 28 May 2025
LayerIF: Estimating Layer Quality for Large Language Models using Influence Functions Hadi Askari Shivanshu Gupta Fei Wang Anshuman Chhabra Muhao Chen TDI 34 0 0 27 May 2025
A Representation Level Analysis of NMT Model Robustness to Grammatical Errors Abderrahmane Issam Yusuf Can Semerci Jan Scholtes Gerasimos Spanakis 25 0 0 27 May 2025
How Syntax Specialization Emerges in Language Models Xufeng Duan Zhaoqian Yao Yunhao Zhang Shaonan Wang Zhenguang G. Cai MILM LRM 61 0 0 26 May 2025
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Hao Chen Haoze Li Zhiqing Xiao Lirong Gao Qi Zhang Xiaomeng Hu Ningtao Wang Xing Fu Junbo Zhao 191 0 0 24 May 2025
Anchored Diffusion Language Model Litu Rout Constantine Caramanis Sanjay Shakkottai 67 0 0 24 May 2025
Multi-Scale Probabilistic Generation Theory: A Hierarchical Framework for Interpreting Large Language Models Yukin Zhang Qi Dong 97 0 0 23 May 2025
SUS backprop: linear backpropagation algorithm for long inputs in transformers Sergey Pankov Georges Harik 91 0 0 21 May 2025
The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation Patrick Kahardipraja Reduan Achtibat Thomas Wiegand Wojciech Samek Sebastian Lapuschkin 143 0 0 21 May 2025
Mechanistic Interpretability of GPT-like Models on Summarization Tasks Anurag Mishra MILM 41 0 0 20 May 2025
$K$ -MSHC: Unmasking Minimally Sufficient Head Circuits in Large Language Models with Experiments on Syntactic Classification Tasks Pratim Chowdhary Peter Chin Deepernab Chakrabarty 90 0 0 18 May 2025
The Way We Prompt: Conceptual Blending, Neural Dynamics, and Prompt-Induced Transitions in LLMs Makoto Sato 35 0 0 16 May 2025
Probability Consistency in Large Language Models: Theoretical Foundations Meet Empirical Discrepancies Xiaoliang Luo Xinyi Xu Michael Ramscar Bradley C. Love 71 0 0 13 May 2025
Are We Paying Attention to Her? Investigating Gender Disambiguation and Attention in Machine Translation Chiara Manna Afra Alishahi Frédéric Blain Eva Vanmassenhove 77 0 0 13 May 2025
LECTOR: Summarizing E-book Reading Content for Personalized Student Support Erwin Daniel López Zapata Cheng Tang Valdemar Švábenský Fumiya Okubo Atsushi Shimada 28 0 0 12 May 2025
CrashSage: A Large Language Model-Centered Framework for Contextual and Interpretable Traffic Crash Analysis Hao Zhen Jidong J. Yang 61 0 0 08 May 2025
Understanding In-context Learning of Addition via Activation Subspaces Xinyan Hu Kayo Yin Michael I. Jordan Jacob Steinhardt Lijie Chen 138 2 0 08 May 2025
Efficient Shapley Value-based Non-Uniform Pruning of Large Language Models Chuan Sun Han Yu Lizhen Cui Xiaoxiao Li 443 3 0 03 May 2025
Fast and Low-Cost Genomic Foundation Models via Outlier Removal Haozheng Luo Chenghao Qiu Maojiang Su Zhihan Zhou Zoe Mehta Guo Ye Jerry Yao-Chieh Hu Han Liu AAML 100 1 0 01 May 2025
Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces Michael Pichat William Pogrund Paloma Pichat Judicael Poumay Armanouche Gasparian Samuel Demarchi Martin Corbet Alois Georgeon Michael Veillet-Guillem MILM 86 0 0 30 Apr 2025
Do Large Language Models know who did what to whom? Joseph M. Denning Xiaohan Bryor Snefjella Idan A. Blank 256 1 0 23 Apr 2025
Exploring How LLMs Capture and Represent Domain-Specific Knowledge Mirian Hipolito Garcia Camille Couturier Daniel Madrigal Diaz Ankur Mallick Anastasios Kyrillidis Robert Sim Victor Rühle Saravan Rajmohan 70 1 0 23 Apr 2025
Word Embedding Techniques for Classification of Star Ratings Hesham Abdelmotaleb Craig McNeile Malgorzata Wojtys 49 0 0 18 Apr 2025
Weight-of-Thought Reasoning: Exploring Neural Network Weights for Enhanced LLM Reasoning Saif Punjwani Larry Heck LRM 93 0 0 14 Apr 2025
Question Tokens Deserve More Attention: Enhancing Large Language Models without Training through Step-by-Step Reading and Question Attention Recalibration Feijiang Han Licheng Guo Hengtao Cui Zhiyuan Lyu LRM 58 0 0 13 Apr 2025
Linguistic Interpretability of Transformer-based Language Models: a systematic review Miguel López-Otal Jorge Gracia Jordi Bernad Carlos Bobed Lucía Pitarch-Ballesteros Emma Anglés-Herrero VLM 106 1 0 09 Apr 2025
Few Dimensions are Enough: Fine-tuning BERT with Selected Dimensions Revealed Its Redundant Nature Shion Fukuhata Yoshinobu Kano 120 0 0 07 Apr 2025
Hallucination Detection using Multi-View Attention Features Yuya Ogasa Yuki Arase 59 0 0 06 Apr 2025
Follow the Flow: On Information Flow Across Textual Tokens in Text-to-Image Models Guy Kaplan Michael Toker Yuval Reif Yonatan Belinkov Roy Schwartz DiffM 105 0 0 01 Apr 2025
Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models Zhanke Zhou Zhaocheng Zhu Xuan Li Mikhail Galkin Xiao Feng Sanmi Koyejo Jian Tang Bo Han LRM 157 6 0 28 Mar 2025
Linguistic Blind Spots of Large Language Models Jiali Cheng Hadi Amiri 99 1 0 25 Mar 2025
Construction Identification and Disambiguation Using BERT: A Case Study of NPN Wesley Scivetti Nathan Schneider 60 0 0 24 Mar 2025
Intra-neuronal attention within language models Relationships between activation and semantics Michael Pichat William Pogrund Paloma Pichat Armanouche Gasparian Samuel Demarchi Corbet Alois Georgeon Michael Veillet-Guillem MILM 89 0 0 17 Mar 2025
Similarity-Aware Token Pruning: Your VLM but Faster Ahmadreza Jeddi Negin Baghbanzadeh Elham Dolatabadi Babak Taati 3DV VLM 111 2 0 14 Mar 2025
Are formal and functional linguistic mechanisms dissociated in language models? Michael Hanna Sandro Pezzelle Yonatan Belinkov 150 1 0 14 Mar 2025
AttentionRAG: Attention-Guided Context Pruning in Retrieval-Augmented Generation Yixiong Fang Tianran Sun Yuling Shi Xiaodong Gu 89 0 0 13 Mar 2025
Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding Seil Kang Jinyeong Kim Junhyeok Kim Seong Jae Hwang VLM 123 4 0 08 Mar 2025
Mitigating Memorization in LLMs using Activation Steering Manan Suri Nishit Anand Amisha Bhaskar LLMSV 117 2 0 08 Mar 2025
(How) Do Language Models Track State? Belinda Z. Li Zifan Carl Guo Jacob Andreas LRM 115 3 0 04 Mar 2025
AxBERT: An Interpretable Chinese Spelling Correction Method Driven by Associative Knowledge Network Fanyu Wang Hangyu Zhu Zhenping Xie 76 0 0 04 Mar 2025