A Survey of Transformers

8 June 2021

Tianyang Lin

Yuxin Wang

Xiangyang Liu

Xipeng Qiu

ViT

ArXiv PDF HTML

Papers citing "A Survey of Transformers"

47 / 347 papers shown

Title
Linear Complexity Randomized Self-attention Mechanism Lin Zheng Chong-Jun Wang Lingpeng Kong 22 31 0 10 Apr 2022
Error Correction Code Transformer Yoni Choukroun Lior Wolf 27 47 0 27 Mar 2022
EmoCaps: Emotion Capsule based Model for Conversational Emotion Recognition Zaijing Li Fengxiao Tang Ming Zhao Yusen Zhu 33 95 0 25 Mar 2022
Transformers Meet Visual Learning Understanding: A Comprehensive Review Yuting Yang Licheng Jiao Xuantong Liu F. Liu Shuyuan Yang Zhixi Feng Xu Tang ViT MedIm 27 28 0 24 Mar 2022
Under the Hood of Transformer Networks for Trajectory Forecasting Luca Franco Leonardo Placidi Francesco Giuliari Irtiza Hasan Marco Cristani Fabio Galasso 10 19 0 22 Mar 2022
FastRPB: a Scalable Relative Positional Encoding for Long Sequence Tasks Maksim Zubkov Daniil Gavrilov 24 0 0 23 Feb 2022
Threats to Pre-trained Language Models: Survey and Taxonomy Shangwei Guo Chunlong Xie Jiwei Li Lingjuan Lyu Tianwei Zhang PILM 27 30 0 14 Feb 2022
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? P. Schramowski Christopher Tauchmann Kristian Kersting FaML 14 87 0 14 Feb 2022
TACTiS: Transformer-Attentional Copulas for Time Series Alexandre Drouin Étienne Marcotte Nicolas Chapados AI4TS 107 37 0 07 Feb 2022
Transformers in Medical Imaging: A Survey Fahad Shamshad Salman Khan Syed Waqas Zamir Muhammad Haris Khan Munawar Hayat F. Khan Huazhu Fu ViT LM&MA MedIm 111 663 0 24 Jan 2022
Video Transformers: A Survey Javier Selva A. S. Johansen Sergio Escalera Kamal Nasrollahi T. Moeslund Albert Clapés ViT 22 103 0 16 Jan 2022
Attention-based Random Forest and Contamination Model Lev V. Utkin A. Konstantinov 26 29 0 08 Jan 2022
Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions Marwan Omar Soohyeon Choi Daehun Nyang David A. Mohaisen 32 57 0 03 Jan 2022
An Ensemble of Pre-trained Transformer Models For Imbalanced Multiclass Malware Classification Ferhat Demirkiran Aykut Çayır U. Ünal Hasan Dag 38 42 0 25 Dec 2021
Contrast and Generation Make BART a Good Dialogue Emotion Recognizer Shimin Li Hang Yan Xipeng Qiu 21 84 0 21 Dec 2021
Trading with the Momentum Transformer: An Intelligent and Interpretable Architecture Kieran Wood Sven Giegerich Stephen J. Roberts S. Zohren AI4TS AIFin 15 21 0 16 Dec 2021
U-shaped Transformer with Frequency-Band Aware Attention for Speech Enhancement Yi Li Yang Sun S. M. Naqvi 20 25 0 11 Dec 2021
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer Zitong Yu Yuming Shen Jingang Shi Hengshuang Zhao Philip H. S. Torr Guoying Zhao ViT MedIm 140 167 0 23 Nov 2021
A Survey of Visual Transformers Yang Liu Yao Zhang Yixin Wang Feng Hou Jin Yuan Jiang Tian Yang Zhang Zhongchao Shi Jianping Fan Zhiqiang He 3DGS ViT 77 330 0 11 Nov 2021
Soft Sensing Transformer: Hundreds of Sensors are Worth a Single Word Chao Zhang Jaswanth K. Yella Yu Huang Xiaoye Qian Sergei Petrov A. Rzhetsky Sthitie Bom 26 14 0 10 Nov 2021
Merging Two Cultures: Deep and Statistical Learning A. Bhadra J. Datta Nicholas G. Polson Vadim Sokolov Jianeng Xu BDL 43 8 0 22 Oct 2021
Evaluating Off-the-Shelf Machine Listening and Natural Language Models for Automated Audio Captioning Benno Weck Xavier Favory K. Drossos Xavier Serra 16 8 0 14 Oct 2021
Towards Efficient NLP: A Standard Evaluation and A Strong Baseline Xiangyang Liu Tianxiang Sun Junliang He Jiawen Wu Lingling Wu Xinyu Zhang Hao Jiang Zhao Cao Xuanjing Huang Xipeng Qiu ELM 28 46 0 13 Oct 2021
Vision Transformer Hashing for Image Retrieval S. Dubey S. Singh Wei Chu ViT 38 49 0 26 Sep 2021
Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning Linyang Li Demin Song Xiaonan Li Jiehang Zeng Ruotian Ma Xipeng Qiu 22 134 0 31 Aug 2021
DeepGene Transformer: Transformer for the gene expression-based classification of cancer subtypes Anwar Khan Boreom Lee MedIm 4 16 0 26 Aug 2021
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing Katikapalli Subramanyam Kalyan A. Rajasekharan S. Sangeetha VLM LM&MA 26 261 0 12 Aug 2021
Attention-like feature explanation for tabular data A. Konstantinov Lev V. Utkin FAtt 21 5 0 10 Aug 2021
How to avoid machine learning pitfalls: a guide for academic researchers M. Lones VLM FaML OnRL 62 77 0 05 Aug 2021
Transformer-based Map Matching Model with Limited Ground-Truth Data using Transfer-Learning Approach Zhixiong Jin Jiwon Kim H. Yeo Seongjin Choi 30 27 0 01 Aug 2021
Dual Aspect Self-Attention based on Transformer for Remaining Useful Life Prediction Zhizheng Zhang Wen Song Qiqiang Li AI4TS 44 165 0 30 Jun 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 40 815 0 14 Jun 2021
Knowledge Inheritance for Pre-trained Language Models Yujia Qin Yankai Lin Jing Yi Jiajie Zhang Xu Han ... Yusheng Su Zhiyuan Liu Peng Li Maosong Sun Jie Zhou VLM 24 49 0 28 May 2021
A Practical Survey on Faster and Lighter Transformers Quentin Fournier G. Caron Daniel Aloise 14 93 0 26 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 289 1,524 0 27 Feb 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 255 4,781 0 24 Feb 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir F. Khan M. Shah ViT 227 2,430 0 04 Jan 2021
A Survey on Deep Reinforcement Learning for Audio-Based Applications S. Latif Heriberto Cuayáhuitl Farrukh Pervez Fahad Shamshad Hafiz Shehbaz Ali Erik Cambria OffRL 44 73 0 01 Jan 2021
ERNIE-Doc: A Retrospective Long-Document Modeling Transformer Siyu Ding Junyuan Shang Shuohuan Wang Yu Sun Hao Tian Hua-Hong Wu Haifeng Wang 71 52 0 31 Dec 2020
Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting Haoyi Zhou Shanghang Zhang J. Peng Shuai Zhang Jianxin Li Hui Xiong Wan Zhang AI4TS 169 3,885 0 14 Dec 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 285 2,015 0 28 Jul 2020
Pre-trained Models for Natural Language Processing: A Survey Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang LM&MA VLM 243 1,452 0 18 Mar 2020
Efficient Content-Based Sparse Attention with Routing Transformers Aurko Roy M. Saffar Ashish Vaswani David Grangier MoE 249 580 0 12 Mar 2020
Talking-Heads Attention Noam M. Shazeer Zhenzhong Lan Youlong Cheng Nan Ding L. Hou 101 80 0 05 Mar 2020
How Much Position Information Do Convolutional Neural Networks Encode? Md. Amirul Islam Sen Jia Neil D. B. Bruce SSL 205 344 0 22 Jan 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
OpenNMT: Open-Source Toolkit for Neural Machine Translation Guillaume Klein Yoon Kim Yuntian Deng Jean Senellart Alexander M. Rush 271 1,896 0 10 Jan 2017