On the Role of Attention Heads in Large Language Model Safety

17 October 2024

Kun Wang

Yang Liu

Sihang Li

Yongbin Li

ArXiv PDF HTML

Papers citing "On the Role of Attention Heads in Large Language Model Safety"

50 / 69 papers shown

Title
ALPS: Attention Localization and Pruning Strategy for Efficient Alignment of Large Language Models Hao Chen Haoze Li Zhiqing Xiao Lirong Gao Qi Zhang Xiaomeng Hu Ningtao Wang Xing Fu Junbo Zhao 172 0 0 24 May 2025
Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Models Yue Li Xin Yi Dongsheng Shi Gerard de Melo Xiaoling Wang Linlin Wang 42 0 0 22 May 2025
ThinkEdit: Interpretable Weight Editing to Mitigate Overly Short Thinking in Reasoning Models Chung-En Sun Ge Yan Tsui-Wei Weng KELM LRM 78 3 0 27 Mar 2025
Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Models Thomas Winninger Boussad Addad Katarzyna Kapusta AAML 102 1 0 08 Mar 2025
Understanding and Rectifying Safety Perception Distortion in VLMs Xiaohan Zou Jian Kang George Kesidis Lu Lin 437 2 0 18 Feb 2025
Reinforced Lifelong Editing for Language Models Zherui Li Houcheng Jiang Hao Chen Baolong Bi Zhenhong Zhou Fei Sun Sihang Li Xinze Wang KELM 104 6 0 09 Feb 2025
Attention Heads of Large Language Models: A Survey Zifan Zheng Yezhaohui Wang Yuxin Huang Shichao Song Mingchuan Yang Bo Tang Feiyu Xiong Zhiyu Li LRM 103 26 0 05 Sep 2024
Extend Model Merging from Fine-Tuned to Pre-Trained Large Language Models via Weight Disentanglement Le Yu Bowen Yu Haiyang Yu Fei Huang Yongbin Li MoMe 65 6 0 06 Aug 2024
Course-Correction: Safety Alignment Using Synthetic Preferences Rongwu Xu Yishuo Cai Zhenhong Zhou Renjie Gu Haiqin Weng Yan Liu Tianwei Zhang Wei Xu Han Qiu 60 7 0 23 Jul 2024
Qwen2 Technical Report An Yang Baosong Yang Binyuan Hui Jian Xu Bowen Yu ... Yuqiong Liu Zeyu Cui Zhenru Zhang Zhifang Guo Zhi-Wei Fan OSLM VLM MU 126 933 0 15 Jul 2024
Transformer Layers as Painters Qi Sun Marc Pickett Aakash Kumar Nain Llion Jones AI4CE 80 18 0 12 Jul 2024
Finding Safety Neurons in Large Language Models Jianhui Chen Xiaozhi Wang Zijun Yao Yushi Bai Lei Hou Juanzi Li KELM LLMSV 63 17 0 20 Jun 2024
Refusal in Language Models Is Mediated by a Single Direction Andy Arditi Oscar Obeso Aaquib Syed Daniel Paleka Nina Panickssery Wes Gurnee Neel Nanda 99 197 0 17 Jun 2024
How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States Zhenhong Zhou Haiyang Yu Xinghua Zhang Rongwu Xu Fei Huang Yongbin Li 76 40 0 09 Jun 2024
Improved Techniques for Optimization-Based Jailbreaking on Large Language Models Xiaojun Jia Tianyu Pang Chao Du Yihao Huang Jindong Gu Yang Liu Xiaochun Cao Min Lin AAML 69 37 0 31 May 2024
Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing Wei Zhao Zhe Li Yige Li Ye Zhang Junfeng Sun KELM AAML 33 32 0 28 May 2024
No Two Devils Alike: Unveiling Distinct Mechanisms of Fine-tuning Attacks Chak Tou Leong Yi Cheng Kaishuai Xu Jian Wang Hanlin Wang Wenjie Li AAML 118 24 0 25 May 2024
Retrieval Head Mechanistically Explains Long-Context Factuality Wenhao Wu Yizhong Wang Guangxuan Xiao Hao-Chun Peng Yao Fu LRM 75 80 0 24 Apr 2024
Mechanistic Interpretability for AI Safety -- A Review Leonard Bereska E. Gavves AI4CE 97 149 0 22 Apr 2024
AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs Zeyi Liao Huan Sun AAML 75 91 0 11 Apr 2024
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models Patrick Chao Edoardo Debenedetti Alexander Robey Maksym Andriushchenko Francesco Croce ... Nicolas Flammarion George J. Pappas F. Tramèr Hamed Hassani Eric Wong ALM ELM AAML 81 121 0 28 Mar 2024
Knowledge Conflicts for LLMs: A Survey Rongwu Xu Zehan Qi Zhijiang Guo Cunxiang Wang Hongru Wang Yue Zhang Wei Xu 246 118 0 13 Mar 2024
Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications Boyi Wei Kaixuan Huang Yangsibo Huang Tinghao Xie Xiangyu Qi Mengzhou Xia Prateek Mittal Mengdi Wang Peter Henderson AAML 91 113 0 07 Feb 2024
On Prompt-Driven Safeguarding for Large Language Models Chujie Zheng Fan Yin Hao Zhou Fandong Meng Jie Zhou Kai-Wei Chang Minlie Huang Nanyun Peng AAML 101 58 0 31 Jan 2024
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs Yi Zeng Hongpeng Lin Jingwen Zhang Diyi Yang Ruoxi Jia Weiyan Shi 78 302 0 12 Jan 2024
A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity Andrew Lee Xiaoyan Bai Itamar Pres Martin Wattenberg Jonathan K. Kummerfeld Rada Mihalcea 95 117 0 03 Jan 2024
Successor Heads: Recurring, Interpretable Attention Heads In The Wild Rhys Gould Euan Ong George Ogden Arthur Conmy LRM 31 49 0 14 Dec 2023
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning Bill Yuchen Lin Abhilasha Ravichander Ximing Lu Nouha Dziri Melanie Sclar Khyathi Chandu Chandra Bhagavatula Yejin Choi 54 190 0 04 Dec 2023
Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patching James Campbell Richard Ren Phillip Guo HILM 47 19 0 25 Nov 2023
Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch Le Yu Yu Bowen Haiyang Yu Fei Huang Yongbin Li MoMe 98 317 0 06 Nov 2023
Managing extreme AI risks amid rapid progress Yoshua Bengio Geoffrey Hinton Andrew Yao Dawn Song Pieter Abbeel ... Philip Torr Stuart J. Russell Daniel Kahneman J. Brauner Sören Mindermann 65 65 0 26 Oct 2023
Safe RLHF: Safe Reinforcement Learning from Human Feedback Josef Dai Xuehai Pan Ruiyang Sun Jiaming Ji Xinbo Xu Mickel Liu Yizhou Wang Yaodong Yang 112 349 0 19 Oct 2023
Jailbreaking Black Box Large Language Models in Twenty Queries Patrick Chao Alexander Robey Yan Sun Hamed Hassani George J. Pappas Eric Wong AAML 105 681 0 12 Oct 2023
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation Yangsibo Huang Samyak Gupta Mengzhou Xia Kai Li Danqi Chen AAML 60 304 0 10 Oct 2023
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! Xiangyu Qi Yi Zeng Tinghao Xie Pin-Yu Chen Ruoxi Jia Prateek Mittal Peter Henderson SILM 109 604 0 05 Oct 2023
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models Xiaogeng Liu Nan Xu Muhao Chen Chaowei Xiao SILM 77 314 0 03 Oct 2023
Towards Best Practices of Activation Patching in Language Models: Metrics and Methods Fred Zhang Neel Nanda LLMSV 187 109 0 27 Sep 2023
Explainability for Large Language Models: A Survey Haiyan Zhao Hanjie Chen Fan Yang Ninghao Liu Huiqi Deng Hengyi Cai Shuaiqiang Wang Dawei Yin Jundong Li LRM 79 451 0 02 Sep 2023
"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models Xinyue Shen Zhenpeng Chen Michael Backes Yun Shen Yang Zhang SILM 117 282 0 07 Aug 2023
Universal and Transferable Adversarial Attacks on Aligned Language Models Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J. Zico Kolter Matt Fredrikson 291 1,451 0 27 Jul 2023
Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla Tom Lieberum Matthew Rahtz János Kramár Neel Nanda G. Irving Rohin Shah Vladimir Mikulik 87 114 0 18 Jul 2023
Jailbroken: How Does LLM Safety Training Fail? Alexander Wei Nika Haghtalab Jacob Steinhardt 195 958 0 05 Jul 2023
Are aligned neural networks adversarially aligned? Nicholas Carlini Milad Nasr Christopher A. Choquette-Choo Matthew Jagielski Irena Gao ... Pang Wei Koh Daphne Ippolito Katherine Lee Florian Tramèr Ludwig Schmidt AAML 59 245 0 26 Jun 2023
A Simple and Effective Pruning Approach for Large Language Models Mingjie Sun Zhuang Liu Anna Bair J. Zico Kolter 131 416 0 20 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 346 4,298 0 09 Jun 2023
A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis Alessandro Stolfo Yonatan Belinkov Mrinmaya Sachan MILM KELM LRM 78 53 0 24 May 2023
Finding Neurons in a Haystack: Case Studies with Sparse Probing Wes Gurnee Neel Nanda Matthew Pauly Katherine Harvey Dmitrii Troitskii Dimitris Bertsimas MILM 198 213 0 02 May 2023
Towards Automated Circuit Discovery for Mechanistic Interpretability Arthur Conmy Augustine N. Mavor-Parker Aengus Lynch Stefan Heimersheim Adrià Garriga-Alonso 55 314 0 28 Apr 2023
Toxicity in ChatGPT: Analyzing Persona-assigned Language Models Ameet Deshpande Vishvak Murahari Tanmay Rajpurohit Ashwin Kalyan Karthik Narasimhan LM&MA LLMAG 65 360 0 11 Apr 2023
Multi-step Jailbreaking Privacy Attacks on ChatGPT Haoran Li Dadi Guo Wei Fan Mingshi Xu Jie Huang Fanpu Meng Yangqiu Song SILM 91 341 0 11 Apr 2023