Rethinking Skip Connection with Layer Normalization in Transformers and ResNets

15 May 2021

Xuancheng Ren

Yuexian Zou

Papers citing "Rethinking Skip Connection with Layer Normalization in Transformers and ResNets"

22 / 22 papers shown

Title
Understanding the Difficulty of Training Transformers Liyuan Liu Xiaodong Liu Jianfeng Gao Weizhu Chen Jiawei Han AI4CE 60 256 0 17 Apr 2020
ReZero is All You Need: Fast Convergence at Large Depth Thomas C. Bachlechner Bodhisattwa Prasad Majumder H. H. Mao G. Cottrell Julian McAuley AI4CE 83 282 0 10 Mar 2020
Exploring and Distilling Cross-Modal Information for Image Captioning Fenglin Liu Xuancheng Ren Yuanxin Liu Kai Lei Xu Sun ViT 66 52 0 28 Feb 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 142 995 0 12 Feb 2020
MUSE: Parallel Multi-Scale Attention for Sequence to Sequence Learning Guangxiang Zhao Xu Sun Jingjing Xu Zhiyuan Zhang Liangchen Luo LRM 48 49 0 17 Nov 2019
Understanding and Improving Layer Normalization Jingjing Xu Xu Sun Zhiyuan Zhang Guangxiang Zhao Junyang Lin FAtt 95 355 0 16 Nov 2019
Transformers without Tears: Improving the Normalization of Self-Attention Toan Q. Nguyen Julian Salazar 88 231 0 14 Oct 2019
Learning Deep Transformer Models for Machine Translation Qiang Wang Bei Li Tong Xiao Jingbo Zhu Changliang Li Derek F. Wong Lidia S. Chao 80 672 0 05 Jun 2019
Aligning Visual Regions and Textual Concepts for Semantic-Grounded Image Representations Fenglin Liu Yuanxin Liu Xuancheng Ren Xiaodong He Xu Sun VLM 56 82 0 15 May 2019
The Evolved Transformer David R. So Chen Liang Quoc V. Le ViT 110 463 0 30 Jan 2019
Fixup Initialization: Residual Learning Without Normalization Hongyi Zhang Yann N. Dauphin Tengyu Ma ODL AI4CE 91 351 0 27 Jan 2019
Exploiting Deep Representations for Neural Machine Translation Zi-Yi Dou Zhaopeng Tu Xing Wang Shuming Shi Tong Zhang 90 93 0 24 Oct 2018
simNet: Stepwise Image-Topic Merging Network for Generating Detailed and Comprehensive Image Captions Fenglin Liu Xuancheng Ren Yuanxin Liu Houfeng Wang Xu Sun 113 66 0 27 Aug 2018
Tensor2Tensor for Neural Machine Translation Ashish Vaswani Samy Bengio E. Brevdo François Chollet Aidan Gomez ... Nal Kalchbrenner Niki Parmar Ryan Sepassi Noam M. Shazeer Jakob Uszkoreit 94 530 0 16 Mar 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 725 132,199 0 12 Jun 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 413 10,526 0 21 Jul 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 354 10,192 0 16 Mar 2016
Inception-v4, Inception-ResNet and the Impact of Residual Connections on Learning Christian Szegedy Sergey Ioffe Vincent Vanhoucke Alexander A. Alemi 381 14,260 0 23 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.2K 194,322 0 10 Dec 2015
Rethinking the Inception Architecture for Computer Vision Christian Szegedy Vincent Vanhoucke Sergey Ioffe Jonathon Shlens Z. Wojna 3DV BDL 883 27,412 0 02 Dec 2015
Highway Networks R. Srivastava Klaus Greff Jürgen Schmidhuber 181 1,773 0 03 May 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 463 43,328 0 11 Feb 2015