Understanding the Difficulty of Training Transformers

Understanding the Difficulty of Training Transformers

17 April 2020

Xiaodong Liu

Papers citing "Understanding the Difficulty of Training Transformers"

13 / 63 papers shown

Title
ImageNet-21K Pretraining for the Masses T. Ridnik Emanuel Ben-Baruch Asaf Noy Lihi Zelnik-Manor SSeg VLM CLIP 187 690 0 22 Apr 2021
Is BERT a Cross-Disciplinary Knowledge Learner? A Surprising Finding of Pre-trained Models' Transferability Wei-Tsung Kao Hung-yi Lee 16 16 0 12 Mar 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 257 4,816 0 24 Feb 2021
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu Renkun Ni Zheng Xu Kezhi Kong Yifan Jiang Tom Goldstein ODL 41 54 0 16 Feb 2021
Optimizing Deeper Transformers on Small Datasets Peng Xu Dhruv Kumar Wei Yang Wenjie Zi Keyi Tang Chenyang Huang Jackie C.K. Cheung S. Prince Yanshuai Cao AI4CE 24 69 0 30 Dec 2020
Learning Light-Weight Translation Models from Deep Transformer Bei Li Ziyang Wang Hui Liu Quan Du Tong Xiao Chunliang Zhang Jingbo Zhu VLM 120 40 0 27 Dec 2020
On the Transformer Growth for Progressive BERT Training Xiaotao Gu Liyuan Liu Hongkun Yu Jing Li Chong Chen Jiawei Han VLM 69 51 0 23 Oct 2020
Data-Efficient Pretraining via Contrastive Self-Supervision Nils Rethmeier Isabelle Augenstein 36 20 0 02 Oct 2020
A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation Dinghan Shen Ming Zheng Yelong Shen Yanru Qu Weizhu Chen AAML 29 130 0 29 Sep 2020
Very Deep Transformers for Neural Machine Translation Xiaodong Liu Kevin Duh Liyuan Liu Jianfeng Gao 19 102 0 18 Aug 2020
MaxVA: Fast Adaptation of Step Sizes by Maximizing Observed Variance of Gradients Chenfei Zhu Yu Cheng Zhe Gan Furong Huang Jingjing Liu Tom Goldstein ODL 35 2 0 21 Jun 2020
Normalized Attention Without Probability Cage Oliver Richter Roger Wattenhofer 14 21 0 19 May 2020
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 244 350 0 14 Jun 2018