RealFormer: Transformer Likes Residual Attention

21 December 2020

Joshua Ainslie

Papers citing "RealFormer: Transformer Likes Residual Attention"

26 / 26 papers shown

Title
Understanding Why Adam Outperforms SGD: Gradient Heterogeneity in Transformers Akiyoshi Tomihari Issei Sato ODL 61 1 0 31 Jan 2025
Layerwise Recurrent Router for Mixture-of-Experts Zihan Qiu Zeyu Huang Shuang Cheng Yizhi Zhou Zili Wang Ivan Titov Jie Fu MoE 81 2 0 13 Aug 2024
EfficientASR: Speech Recognition Network Compression via Attention Redundancy and Chunk-Level FFN Optimization Jianzong Wang Ziqi Liang Xulong Zhang Ning Cheng Jing Xiao 38 0 0 30 Apr 2024
Convolution-enhanced Evolving Attention Networks Yujing Wang Yaming Yang Zhuowan Li Jiangang Bai Mingliang Zhang Xiangtai Li Jiahao Yu Ce Zhang Gao Huang Yu Tong ViT 27 6 0 16 Dec 2022
Paraphrase Identification with Deep Learning: A Review of Datasets and Methods Chao Zhou Cheng Qiu Daniel Ernesto Acuna 32 25 0 13 Dec 2022
TencentPretrain: A Scalable and Flexible Toolkit for Pre-training Models of Different Modalities Zhe Zhao Yudong Li Cheng-An Hou Jing-xin Zhao Rong Tian ... Xingwu Sun Zhanhui Kang Xiaoyong Du Linlin Shen Kimmo Yan VLM 41 23 0 13 Dec 2022
Uncertainty-aware Vision-based Metric Cross-view Geolocalization F. Fervers Sebastian Bullinger C. Bodensteiner Michael Arens Rainer Stiefelhagen 29 39 0 22 Nov 2022
Relational Graph Convolutional Neural Networks for Multihop Reasoning: A Comparative Study Ieva Staliunaite P. Gorinski Ignacio Iacobacci GNN 27 0 0 12 Oct 2022
GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng Xiao Liu Zhengxiao Du Zihan Wang Hanyu Lai ... Jidong Zhai Wenguang Chen Peng-Zhen Zhang Yuxiao Dong Jie Tang BDL LRM 253 1,073 0 05 Oct 2022
Searching a High-Performance Feature Extractor for Text Recognition Network Hui Zhang Quanming Yao James T. Kwok X. Bai 28 7 0 27 Sep 2022
Improving Transformer-based Conversational ASR by Inter-Sentential Attention Mechanism Kun Wei Pengcheng Guo Ning Jiang 48 11 0 02 Jul 2022
EATFormer: Improving Vision Transformer Inspired by Evolutionary Algorithm Jiangning Zhang Xiangtai Li Yabiao Wang Chengjie Wang Yibo Yang Yong Liu Dacheng Tao ViT 34 32 0 19 Jun 2022
Revisiting Over-smoothing in BERT from the Perspective of Graph Han Shi Jiahui Gao Hang Xu Xiaodan Liang Zhenguo Li Lingpeng Kong Stephen M. S. Lee James T. Kwok 22 71 0 17 Feb 2022
Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks Qingsong Lv Ming Ding Qiang Liu Yuxiang Chen Wenzheng Feng Siming He Chang Zhou Jianguo Jiang Yuxiao Dong Jie Tang 42 312 0 30 Dec 2021
TRIG: Transformer-Based Text Recognizer with Initial Embedding Guidance Yuefeng Tao Zhiwei Jia Runze Ma Shugong Xu ViT 19 6 0 16 Nov 2021
MNet-Sim: A Multi-layered Semantic Similarity Network to Evaluate Sentence Similarity Manuela Nayantara Jeyaraj D. Kasthurirathna 11 3 0 09 Nov 2021
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer Yifan Xu Zhijie Zhang Mengdan Zhang Kekai Sheng Ke Li Weiming Dong Liqing Zhang Changsheng Xu Xing Sun ViT 32 201 0 03 Aug 2021
MedGPT: Medical Concept Prediction from Clinical Narratives Z. Kraljevic Anthony Shek D. Bean R. Bendayan J. Teo Richard J. B. Dobson LM&MA AI4TS MedIm 25 39 0 07 Jul 2021
Attention-based multi-channel speaker verification with ad-hoc microphone arrays Che-Yuan Liang Junqi Chen Shanzheng Guan Xiao-Lei Zhang 20 9 0 01 Jul 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 53 1,088 0 08 Jun 2021
Analogous to Evolutionary Algorithm: Designing a Unified Sequence Model Jiangning Zhang Chao Xu Jian Li Wenzhou Chen Yabiao Wang Ying Tai Shuo Chen Chengjie Wang Feiyue Huang Yong Liu 32 22 0 31 May 2021
VARA-TTS: Non-Autoregressive Text-to-Speech Synthesis based on Very Deep VAE with Residual Attention Peng Liu Yuewen Cao Songxiang Liu Na Hu Guangzhi Li Chao Weng Dan Su 39 22 0 12 Feb 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,017 0 28 Jul 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 252 580 0 12 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,826 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018