Transformer Feed-Forward Layers Are Key-Value Memories

29 December 2020

Papers citing "Transformer Feed-Forward Layers Are Key-Value Memories"

50 / 151 papers shown

Title
AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection Kai Hua Steven Wu Ge Zhang Ke Shen LRM 28 0 0 12 May 2025
Pre-training vs. Fine-tuning: A Reproducibility Study on Dense Retrieval Knowledge Acquisition Zheng Yao Shuai Wang Guido Zuccon 21 0 0 12 May 2025
UMoE: Unifying Attention and FFN with Shared Experts Yuanhang Yang Chaozheng Wang Jing Li MoE 29 0 0 12 May 2025
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models Isaac Gerber 31 0 0 10 May 2025
Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classification Leon Eshuijs Shihan Wang Antske Fokkens 26 0 0 09 May 2025
FloE: On-the-Fly MoE Inference on Memory-constrained GPU Yuxin Zhou Zheng Li J. Zhang Jue Wang Yunhong Wang Zhongle Xie Ke Chen Lidan Shou MoE 52 0 0 09 May 2025
Defending against Indirect Prompt Injection by Instruction Detection Tongyu Wen Chenglong Wang Xiyuan Yang Haoyu Tang Yueqi Xie Lingjuan Lyu Zhicheng Dou Fangzhao Wu AAML 31 0 0 08 May 2025
Scalable LLM Math Reasoning Acceleration with Low-rank Distillation Harry Dong Bilge Acun Beidi Chen Yuejie Chi LRM 34 0 0 08 May 2025
Demystifying optimized prompts in language models Rimon Melamed Lucas H. McCabe H. H. Huang 39 0 0 04 May 2025
Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation Vaidehi Patil Yi-Lin Sung Peter Hase Jie Peng Jen-tse Huang Joey Tianyi Zhou AAML MU 83 3 0 01 May 2025
On the generalization of language models from in-context learning and finetuning: a controlled study Andrew Kyle Lampinen Arslan Chaudhry Stephanie Chan Cody Wild Diane Wan Alex Ku Jorg Bornschein Razvan Pascanu Murray Shanahan James L. McClelland 46 0 0 01 May 2025
SetKE: Knowledge Editing for Knowledge Elements Overlap Yifan Wei Xiaoyan Yu Ran Song Hao Peng Angsheng Li KELM 62 0 0 29 Apr 2025
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference Zhenyu (Allen) Zhang Zechun Liu Yuandong Tian Harshit Khaitan Zhilin Wang Steven Li 57 0 0 28 Apr 2025
Revisiting Transformers through the Lens of Low Entropy and Dynamic Sparsity Ruifeng Ren Yong Liu 132 0 0 26 Apr 2025
Param $Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost Sheng Cao Mingrui Wu Karthik Prasad Yuandong Tian Zechun Liu MoMe 80 0 0 23 Apr 2025
Bigram Subnetworks: Mapping to Next Tokens in Transformer Language Models Tyler A. Chang Benjamin Bergen 50 0 0 21 Apr 2025
Signatures of human-like processing in Transformer forward passes Jennifer Hu Michael A. Lepori Michael Franke AI4CE 156 0 0 18 Apr 2025
Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric Yixin Cao Jiahao Ying Yixuan Wang Xipeng Qiu Xuanjing Huang Yugang Jiang ELM 41 2 0 10 Apr 2025
Steering off Course: Reliability Challenges in Steering Language Models Patrick Queiroz Da Silva Hari Sethuraman Dheeraj Rajagopal Hannaneh Hajishirzi Sachin Kumar LLMSV 29 1 0 06 Apr 2025
Effective Skill Unlearning through Intervention and Abstention Yongce Li Chung-En Sun Tsui-Wei Weng MU 157 0 0 27 Mar 2025
BioMamba: Leveraging Spectro-Temporal Embedding in Bidirectional Mamba for Enhanced Biosignal Classification Jian Qian Teck Lun Goh Bingyu Xie Chengyao Zhu Biao Wan Yawen Guan Rachel Ding Chen Patrick Chiang Mamba 47 0 0 14 Mar 2025
Are formal and functional linguistic mechanisms dissociated in language models? Michael Hanna Sandro Pezzelle Yonatan Belinkov 47 0 0 14 Mar 2025
Discovering Influential Neuron Path in Vision Transformers Yifan Wang Yifei Liu Yingdong Shi Chong Li Anqi Pang Sibei Yang Jingyi Yu Kan Ren ViT 69 0 0 12 Mar 2025
Exploiting Edited Large Language Models as General Scientific Optimizers Qitan Lv T. Liu Haoyu Wang 41 0 0 08 Mar 2025
Strategy Coopetition Explains the Emergence and Transience of In-Context Learning Aaditya K. Singh Ted Moskovitz Sara Dragutinovic Felix Hill Stephanie C. Y. Chan Andrew Saxe 145 0 0 07 Mar 2025
SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction Lu Dai Yijie Xu Jinhui Ye Hao Liu Hui Xiong 3DV RALM 83 2 0 03 Mar 2025
Triple Phase Transitions: Understanding the Learning Dynamics of Large Language Models from a Neuroscience Perspective Yuko Nakagi Keigo Tada Sota Yoshino Shinji Nishimoto Yu Takagi LRM 37 0 0 28 Feb 2025
Finite State Automata Inside Transformers with Chain-of-Thought: A Mechanistic Study on State Tracking Yifan Zhang Wenyu Du Dongming Jin Jie Fu Zhi Jin LRM 53 0 0 27 Feb 2025
Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utility Martin Kuo Jingyang Zhang Jianyi Zhang Minxue Tang Louis DiValentin ... William Chen Amin Hass Tianlong Chen Y. Chen Hao Li MU KELM 51 2 0 24 Feb 2025
Revealing and Mitigating Over-Attention in Knowledge Editing Pinzheng Wang Zecheng Tang Keyan Zhou J. Li Qiaoming Zhu M. Zhang KELM 120 2 0 21 Feb 2025
Repetition Neurons: How Do Language Models Produce Repetitions? Tatsuya Hiraoka Kentaro Inui MILM 75 6 0 21 Feb 2025
An LLM-based Agent for Reliable Docker Environment Configuration Ruida Hu Chao Peng Xinchen Wang Cuiyun Gao 78 2 0 19 Feb 2025
The Knowledge Microscope: Features as Better Analytical Lenses than Neurons Yuheng Chen Pengfei Cao Kang Liu Jun Zhao 50 0 0 18 Feb 2025
DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs Minxuan Lv Zhenpeng Su Leiyu Pan Yizhe Xiong Zijia Lin ... Guiguang Ding Cheng Luo Di Zhang Kun Gai Songlin Hu MoE 41 0 0 18 Feb 2025
DELMAN: Dynamic Defense Against Large Language Model Jailbreaking with Model Editing Yi Wang Fenghua Weng Songlin Yang Zhan Qin Minlie Huang Wenjie Wang KELM AAML 53 0 0 17 Feb 2025
Mechanistic Unveiling of Transformer Circuits: Self-Influence as a Key to Model Reasoning L. Zhang Lijie Hu Di Wang LRM 95 0 0 17 Feb 2025
AnyEdit: Edit Any Knowledge Encoded in Language Models Houcheng Jiang Fan Zhang Ningyu Zhang Guojun Ma Mingyang Wan Xuben Wang Xiangnan He Tat-Seng Chua KELM 57 8 0 08 Feb 2025
Understanding and Mitigating Gender Bias in LLMs via Interpretable Neuron Editing Zeping Yu Sophia Ananiadou KELM 43 1 0 24 Jan 2025
Transforming Vision Transformer: Towards Efficient Multi-Task Asynchronous Learning Hanwen Zhong Jiaxin Chen Yutong Zhang Di Huang Yunhong Wang MoE 42 0 0 12 Jan 2025
Key-value memory in the brain Samuel J. Gershman Ila Fiete Kazuki Irie 34 7 0 06 Jan 2025
Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention Zhenyu Guo Wenguang Chen 46 0 0 01 Jan 2025
Out-of-distribution generalization via composition: a lens through induction heads in Transformers Jiajun Song Zhuoyan Xu Yiqiao Zhong 85 4 0 31 Dec 2024
How Do Artificial Intelligences Think? The Three Mathematico-Cognitive Factors of Categorical Segmentation Operated by Synthetic Neurons Michael Pichat William Pogrund Armanush Gasparian Paloma Pichat Samuel Demarchi Michael Veillet-Guillem 42 3 0 26 Dec 2024
Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection Le Yang Ziwei Zheng Boxu Chen Zhengyu Zhao Chenhao Lin Chao Shen VLM 140 3 0 18 Dec 2024
Bridging the Visual Gap: Fine-Tuning Multimodal Models with Knowledge-Adapted Captions Moran Yanuka Assaf Ben-Kish Yonatan Bitton Idan Szpektor Raja Giryes VLM 47 2 0 13 Nov 2024
Controllable Context Sensitivity and the Knob Behind It Julian Minder Kevin Du Niklas Stoehr Giovanni Monea Chris Wendler Robert West Ryan Cotterell KELM 55 3 0 11 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 49 2 0 30 Oct 2024
WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models Jinghan Jia Jiancheng Liu Yihua Zhang Parikshit Ram Nathalie Baracaldo Sijia Liu MU 35 2 0 23 Oct 2024
Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering Yu Zhao Alessio Devoto Giwon Hong Xiaotang Du Aryo Pradipta Gema Hongru Wang Xuanli He Kam-Fai Wong Pasquale Minervini KELM LLMSV 36 16 0 21 Oct 2024
ReDeEP: Detecting Hallucination in Retrieval-Augmented Generation via Mechanistic Interpretability ZhongXiang Sun Xiaoxue Zang Kai Zheng Yang Song Jun Xu Xiao Zhang Weijie Yu Yang Song Han Li 57 7 0 15 Oct 2024