Augmenting Self-attention with Persistent Memory

2 July 2019

Papers citing "Augmenting Self-attention with Persistent Memory"

35 / 35 papers shown

Title
Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models Isaac Gerber 34 0 0 10 May 2025
Human-inspired Perspectives: A Survey on AI Long-term Memory Zihong He Weizhe Lin Hao Zheng Fan Zhang Matt Jones Laurence Aitchison X. Xu Miao Liu Per Ola Kristensson Junxiao Shen 79 2 0 01 Nov 2024
TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters Haiyang Wang Yue Fan Muhammad Ferjad Naeem Yongqin Xian J. E. Lenssen Liwei Wang F. Tombari Bernt Schiele 49 2 0 30 Oct 2024
Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory Xueyan Niu Bo Bai Lei Deng Wei Han 44 6 0 14 May 2024
Memory Mosaics Jianyu Zhang Niklas Nolte Ranajoy Sadhukhan Beidi Chen Léon Bottou VLM 73 3 0 10 May 2024
Goodtriever: Adaptive Toxicity Mitigation with Retrieval-augmented Models Luiza Amador Pozzobon Beyza Ermis Patrick Lewis Sara Hooker 36 20 0 11 Oct 2023
Think before you speak: Training Language Models With Pause Tokens Sachin Goyal Ziwei Ji A. S. Rawat A. Menon Sanjiv Kumar Vaishnavh Nagarajan LRM 29 97 0 03 Oct 2023
Transformer-VQ: Linear-Time Transformers via Vector Quantization Albert Mohwald 36 15 0 28 Sep 2023
With a Little Help from your own Past: Prototypical Memory Networks for Image Captioning Manuele Barraco Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara VLM 60 19 0 23 Aug 2023
Towards A Unified View of Sparse Feed-Forward Network in Pretraining Large Language Model Leo Liu Tim Dettmers Xi Lin Ves Stoyanov Xian Li MoE 26 9 0 23 May 2023
A Study on ReLU and Softmax in Transformer Kai Shen Junliang Guo Xuejiao Tan Siliang Tang Rui Wang Jiang Bian 29 53 0 13 Feb 2023
Interpretability in Activation Space Analysis of Transformers: A Focused Survey Soniya Vijayakumar AI4CE 40 3 0 22 Jan 2023
Bird-Eye Transformers for Text Generation Models Lei Sha Yuhang Song Yordan Yordanov Tommaso Salvatori Thomas Lukasiewicz 30 0 0 08 Oct 2022
Analyzing Transformers in Embedding Space Guy Dar Mor Geva Ankit Gupta Jonathan Berant 29 84 0 06 Sep 2022
Momentum Transformer: Closing the Performance Gap Between Self-attention and Its Linearization T. Nguyen Richard G. Baraniuk Robert M. Kirby Stanley J. Osher Bao Wang 42 9 0 01 Aug 2022
Exploring the sequence length bottleneck in the Transformer for Image Captioning Jiapeng Hu Roberto Cavicchioli Alessandro Capotondi ViT 38 3 0 07 Jul 2022
Plug-and-Play Adaptation for Continuously-updated QA Kyungjae Lee Wookje Han Seung-won Hwang Hwaran Lee Joonsuk Park Sang-Woo Lee KELM 30 16 0 27 Apr 2022
Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space Mor Geva Avi Caciularu Ke Wang Yoav Goldberg KELM 74 338 0 28 Mar 2022
ANNA: Enhanced Language Representation for Question Answering Changwook Jun Hansol Jang Myoseop Sim Hyun Kim Jooyoung Choi Kyungkoo Min Kyunghoon Bae 31 6 0 28 Mar 2022
Memorizing Transformers Yuhuai Wu M. Rabe DeLesley S. Hutchins Christian Szegedy RALM 30 173 0 16 Mar 2022
CaMEL: Mean Teacher Learning for Image Captioning Manuele Barraco Matteo Stefanini Marcella Cornia S. Cascianelli Lorenzo Baraldi Rita Cucchiara ViT VLM 43 27 0 21 Feb 2022
The Dual Form of Neural Networks Revisited: Connecting Test Time Predictions to Training Patterns via Spotlights of Attention Kazuki Irie Róbert Csordás Jürgen Schmidhuber 19 42 0 11 Feb 2022
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling Renrui Zhang Rongyao Fang Wei Zhang Peng Gao Kunchang Li Jifeng Dai Yu Qiao Hongsheng Li VLM 196 387 0 06 Nov 2021
Layer-wise Pruning of Transformer Attention Heads for Efficient Language Modeling Kyuhong Shim Iksoo Choi Wonyong Sung Jungwook Choi 40 15 0 07 Oct 2021
Combining Transformers with Natural Language Explanations Federico Ruggeri Marco Lippi Paolo Torroni 25 1 0 02 Sep 2021
Lyapunov-based uncertainty-aware safe reinforcement learning Ashkan B. Jeddi Nariman L. Dehghani A. Shafieezadeh 14 7 0 29 Jul 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 53 1,089 0 08 Jun 2021
Mask Attention Networks: Rethinking and Strengthen Transformer Zhihao Fan Yeyun Gong Dayiheng Liu Zhongyu Wei Siyuan Wang Jian Jiao Nan Duan Ruofei Zhang Xuanjing Huang 34 72 0 25 Mar 2021
Transformer Feed-Forward Layers Are Key-Value Memories Mor Geva R. Schuster Jonathan Berant Omer Levy KELM 39 754 0 29 Dec 2020
Efficient Transformers: A Survey Yi Tay Mostafa Dehghani Dara Bahri Donald Metzler VLM 114 1,104 0 14 Sep 2020
Simplified Self-Attention for Transformer-based End-to-End Speech Recognition Haoneng Luo Shiliang Zhang Ming Lei Lei Xie 35 33 0 21 May 2020
Lite Transformer with Long-Short Range Attention Zhanghao Wu Zhijian Liu Ji Lin Chengyue Wu Song Han 23 318 0 24 Apr 2020
Meshed-Memory Transformer for Image Captioning Marcella Cornia Matteo Stefanini Lorenzo Baraldi Rita Cucchiara 14 869 0 17 Dec 2019
DFSMN-SAN with Persistent Memory Model for Automatic Speech Recognition Zhao You Dan Su Jie Chen Chao Weng Dong Yu 28 13 0 28 Oct 2019
Transformers without Tears: Improving the Normalization of Self-Attention Toan Q. Nguyen Julian Salazar 50 225 0 14 Oct 2019