v1v2 (latest)

AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks

18 September 2021

G. Bingham

Risto Miikkulainen

ODL

ArXiv (abs)PDF HTML Github (27★)

Papers citing "AutoInit: Analytic Signal-Preserving Weight Initialization for Neural Networks"

50 / 59 papers shown

Title
ResNet strikes back: An improved training procedure in timm Ross Wightman Hugo Touvron Hervé Jégou AI4TS 268 499 0 01 Oct 2021
CoAtNet: Marrying Convolution and Attention for All Data Sizes Zihang Dai Hanxiao Liu Quoc V. Le Mingxing Tan ViT 141 1,212 0 09 Jun 2021
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu Renkun Ni Zheng Xu Kezhi Kong Wenjie Huang Tom Goldstein ODL 101 56 0 16 Feb 2021
Characterizing signal propagation to close the performance gap in unnormalized ResNets Andrew Brock Soham De Samuel L. Smith 178 123 0 21 Jan 2021
Smooth Adversarial Training Cihang Xie Mingxing Tan Boqing Gong Alan Yuille Quoc V. Le OOD 80 154 0 25 Jun 2020
Array Programming with NumPy Charles R. Harris K. Millman S. Walt R. Gommers Pauli Virtanen ... Tyler Reddy Warren Weckesser Hameer Abbasi C. Gohlke T. Oliphant 156 15,026 0 18 Jun 2020
Discovering Parametric Activation Functions G. Bingham Risto Miikkulainen ODL 63 71 0 05 Jun 2020
A survey on modern trainable activation functions Andrea Apicella Francesco Donnarumma Francesco Isgrò R. Prevete 101 378 0 02 May 2020
Evolving Normalization-Activation Layers Hanxiao Liu Andrew Brock Karen Simonyan Quoc V. Le 106 81 0 06 Apr 2020
ReZero is All You Need: Fast Convergence at Large Depth Thomas C. Bachlechner Bodhisattwa Prasad Majumder H. H. Mao G. Cottrell Julian McAuley AI4CE 85 282 0 10 Mar 2020
Evolutionary Optimization of Deep Learning Activation Functions G. Bingham William Macke Risto Miikkulainen ODL 48 50 0 17 Feb 2020
Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks Wei Hu Lechao Xiao Jeffrey Pennington 74 115 0 16 Jan 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 559 42,639 0 03 Dec 2019
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 260 3,505 0 30 Sep 2019
How to Initialize your Network? Robust Initialization for WeightNorm & ResNets Devansh Arpit Victor Campos Yoshua Bengio 64 59 0 05 Jun 2019
On Network Design Spaces for Visual Recognition Ilija Radosavovic Justin Johnson Saining Xie Wan-Yen Lo Piotr Dollár 88 136 0 30 May 2019
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features Sangdoo Yun Dongyoon Han Seong Joon Oh Sanghyuk Chun Junsuk Choe Y. Yoo OOD 624 4,809 0 13 May 2019
A Survey on Neural Architecture Search Martin Wistuba Ambrish Rawat Tejaswini Pedapati AI4CE 77 259 0 04 May 2019
Evolutionary Neural AutoML for Deep Learning J. Liang Elliot Meyerson Babak Hodjat Daniel Fink Karl Mutch Risto Miikkulainen 48 108 0 18 Feb 2019
Fixup Initialization: Residual Learning Without Normalization Hongyi Zhang Yann N. Dauphin Tengyu Ma ODL AI4CE 100 351 0 27 Jan 2019
Activation Functions: Comparison of trends in Practice and Research for Deep Learning S. Bodenstedt Dominik Rivoir A. Gachagan S. T. Mees 103 1,279 0 08 Nov 2018
The Quest for the Golden Activation Function Mina Basirat P. Roth 67 53 0 02 Aug 2018
Glow: Generative Flow with Invertible 1x1 Convolutions Diederik P. Kingma Prafulla Dhariwal BDL DRL 305 3,144 0 09 Jul 2018
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks Minmin Chen Jeffrey Pennington S. Schoenholz SyDa AI4CE 67 117 0 14 Jun 2018
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 311 354 0 14 Jun 2018
Group Normalization Yuxin Wu Kaiming He 247 3,676 0 22 Mar 2018
How to Start Training: The Effect of Initialization and Architecture Boris Hanin David Rolnick 91 256 0 05 Mar 2018
Which Neural Net Architectures Give Rise To Exploding and Vanishing Gradients? Boris Hanin 85 256 0 11 Jan 2018
Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice Jeffrey Pennington S. Schoenholz Surya Ganguli ODL 45 254 0 13 Nov 2017
Three Factors Influencing Minima in SGD Stanislaw Jastrzebski Zachary Kenton Devansh Arpit Nicolas Ballas Asja Fischer Yoshua Bengio Amos Storkey 78 463 0 13 Nov 2017
Don't Decay the Learning Rate, Increase the Batch Size Samuel L. Smith Pieter-Jan Kindermans Chris Ying Quoc V. Le ODL 114 996 0 01 Nov 2017
mixup: Beyond Empirical Risk Minimization Hongyi Zhang Moustapha Cissé Yann N. Dauphin David Lopez-Paz NoLa 316 9,811 0 25 Oct 2017
Deep Residual Networks and Weight Initialization Masato Taki ODL 50 24 0 09 Sep 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017
Self-Normalizing Neural Networks Günter Klambauer Thomas Unterthiner Andreas Mayr Sepp Hochreiter 486 2,520 0 08 Jun 2017
Evolving Deep Neural Networks Risto Miikkulainen J. Liang Elliot Meyerson Aditya Rawal Daniel Fink ... B. Raju Hormoz Shahrzad Arshak Navruzyan Nigel P. Duffy Babak Hodjat 108 891 0 01 Mar 2017
PMLB: A Large Benchmark Suite for Machine Learning Evaluation and Comparison Randal S. Olson William La Cava Patryk Orzechowski Ryan J. Urbanowicz J. Moore 412 381 0 01 Mar 2017
Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning Stefan Elfwing E. Uchibe Kenji Doya 141 1,746 0 10 Feb 2017
Ex Machina: Personal Attacks Seen at Scale Ellery Wulczyn Nithum Thain Lucas Dixon 125 762 0 27 Oct 2016
Densely Connected Convolutional Networks Gao Huang Zhuang Liu Laurens van der Maaten Kilian Q. Weinberger PINN 3DV 822 36,892 0 25 Aug 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 350 8,179 0 13 Aug 2016
Layer Normalization Jimmy Lei Ba J. Kiros Geoffrey E. Hinton 435 10,541 0 21 Jul 2016
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 174 5,049 0 27 Jun 2016
TensorFlow: A system for large-scale machine learning Martín Abadi P. Barham Jianmin Chen Zhiwen Chen Andy Davis ... Vijay Vasudevan Pete Warden Martin Wicke Yuan Yu Xiaoqiang Zhang GNN AI4CE 435 18,361 0 27 May 2016
Wide Residual Networks Sergey Zagoruyko N. Komodakis 356 8,002 0 23 May 2016
Identity Mappings in Deep Residual Networks Kaiming He Xinming Zhang Shaoqing Ren Jian Sun 354 10,196 0 16 Mar 2016
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Tim Salimans Diederik P. Kingma ODL 196 1,945 0 25 Feb 2016
Deep Residual Learning for Image Recognition Kaiming He Xinming Zhang Shaoqing Ren Jian Sun MedIm 2.3K 194,510 0 10 Dec 2015
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) Djork-Arné Clevert Thomas Unterthiner Sepp Hochreiter 307 5,536 0 23 Nov 2015
Data-dependent Initializations of Convolutional Neural Networks Philipp Krahenbuhl Carl Doersch Jeff Donahue Trevor Darrell VLM 95 203 0 21 Nov 2015