The effect of Target Normalization and Momentum on Dying ReLU

13 May 2020

Danica Kragic

Papers citing "The effect of Target Normalization and Momentum on Dying ReLU"

20 / 20 papers shown

Title
Why ReLU Units Sometimes Die: Analysis of Single-Unit Error Backpropagation in Neural Networks S. Douglas Jiutian Yu 15 45 0 14 Dec 2018
Gradient Descent Finds Global Minima of Deep Neural Networks S. Du Jason D. Lee Haochuan Li Liwei Wang Masayoshi Tomizuka ODL 105 1,133 0 09 Nov 2018
ANS: Adaptive Network Scaling for Deep Rectifier Reinforcement Learning Models Yueh-hua Wu Fan-Yun Sun Yen-Yu Chang Shou-De Lin 30 5 0 06 Sep 2018
How Does Batch Normalization Help Optimization? Shibani Santurkar Dimitris Tsipras Andrew Ilyas Aleksander Madry ODL 60 1,531 0 29 May 2018
Deep Learning using Rectified Linear Units (ReLU) Abien Fred Agarap 49 3,195 0 22 Mar 2018
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients? Boris Hanin 34 254 0 11 Jan 2018
Searching for Activation Functions Prajit Ramachandran Barret Zoph Quoc V. Le 44 602 0 16 Oct 2017
Deep Reinforcement Learning that Matters Peter Henderson Riashat Islam Philip Bachman Joelle Pineau Doina Precup David Meger OffRL 90 1,940 0 19 Sep 2017
Self-Normalizing Neural Networks Günter Klambauer Thomas Unterthiner Andreas Mayr Sepp Hochreiter 202 2,496 0 08 Jun 2017
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 204 10,412 0 21 Jul 2016
Exponential expressivity in deep neural networks through transient chaos Ben Poole Subhaneil Lahiri M. Raghu Jascha Narain Sohl-Dickstein Surya Ganguli 71 587 0 16 Jun 2016
Parametric Exponential Linear Unit for Deep Convolutional Neural Networks Ludovic Trottier Philippe Giguère B. Chaib-draa 48 199 0 30 May 2016
Learning values across many orders of magnitude H. V. Hasselt A. Guez Matteo Hessel Volodymyr Mnih David Silver 34 170 0 24 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 1.1K 192,638 0 10 Dec 2015
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter 184 5,502 0 23 Nov 2015
Training Very Deep Networks R. Srivastava Klaus Greff Jürgen Schmidhuber 81 1,675 0 22 Jul 2015
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift Sergey Ioffe Christian Szegedy OOD 228 43,154 0 11 Feb 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 109 18,534 0 06 Feb 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 535 149,474 0 22 Dec 2014
Deeply learned face representations are sparse, selective, and robust Yi Sun Xiaogang Wang Xiaoou Tang CVBM 304 921 0 03 Dec 2014