Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets

24 October 2020

Papers citing "Inductive Bias of Gradient Descent for Weight Normalized Smooth Homogeneous Neural Nets"

27 / 27 papers shown

Title
Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy E. Moroshko Suriya Gunasekar Blake E. Woodworth Jason D. Lee Nathan Srebro Daniel Soudry 59 85 0 13 Jul 2020
Optimization Theory for ReLU Neural Networks Trained with Normalization Layers Yonatan Dukler Quanquan Gu Guido Montúfar 56 30 0 11 Jun 2020
Directional convergence and alignment in deep learning Ziwei Ji Matus Telgarsky 52 171 0 11 Jun 2020
Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Loss Lénaïc Chizat Francis R. Bach MLT 106 338 0 11 Feb 2020
Rethinking Normalization and Elimination Singularity in Neural Networks Siyuan Qiao Huiyu Wang Chenxi Liu Wei Shen Alan Yuille 41 10 0 21 Nov 2019
Implicit Regularization and Convergence for Weight Normalization Xiaoxia Wu Yan Sun Zhaolin Ren Shanshan Wu Zhiyuan Li Suriya Gunasekar Rachel A. Ward Qiang Liu 125 21 0 18 Nov 2019
Positional Normalization Boyi Li Felix Wu Kilian Q. Weinberger Serge J. Belongie 49 91 0 09 Jul 2019
Gradient Descent Maximizes the Margin of Homogeneous Neural Networks Kaifeng Lyu Jian Li 75 334 0 13 Jun 2019
Characterizing the implicit bias via a primal-dual analysis Ziwei Ji Matus Telgarsky 41 20 0 11 Jun 2019
Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models Mor Shpigel Nacson Suriya Gunasekar Jason D. Lee Nathan Srebro Daniel Soudry 60 92 0 17 May 2019
Theoretical Analysis of Auto Rate-Tuning by Batch Normalization Sanjeev Arora Zhiyuan Li Kaifeng Lyu 72 131 0 10 Dec 2018
Gradient descent aligns the layers of deep linear networks Ziwei Ji Matus Telgarsky 113 254 0 04 Oct 2018
Towards Understanding Regularization in Batch Normalization Ping Luo Xinjiang Wang Wenqi Shao Zhanglin Peng MLT AI4CE 53 180 0 04 Sep 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 250 3,191 0 20 Jun 2018
Implicit Bias of Gradient Descent on Linear Convolutional Networks Suriya Gunasekar Jason D. Lee Daniel Soudry Nathan Srebro MDE 105 411 0 01 Jun 2018
Understanding Batch Normalization Johan Bjorck Carla P. Gomes B. Selman Kilian Q. Weinberger 140 610 0 01 Jun 2018
How Does Batch Normalization Help Optimization? Shibani Santurkar Dimitris Tsipras Andrew Ilyas Aleksander Madry ODL 92 1,537 0 29 May 2018
Exponential convergence rates for Batch Normalization: The power of length-direction decoupling in non-convex optimization Jonas Köhler Hadi Daneshmand Aurelien Lucchi M. Zhou K. Neymeyr Thomas Hofmann 51 91 0 27 May 2018
Bilinear Attention Networks Jin-Hwa Kim Jaehyun Jun Byoung-Tak Zhang AIMat 84 873 0 21 May 2018
Convergence of Gradient Descent on Separable Data Mor Shpigel Nacson Jason D. Lee Suriya Gunasekar Pedro H. P. Savarese Nathan Srebro Daniel Soudry 65 169 0 05 Mar 2018
Characterizing Implicit Bias in Terms of Optimization Geometry Suriya Gunasekar Jason D. Lee Daniel Soudry Nathan Srebro AI4CE 71 409 0 22 Feb 2018
Sockeye: A Toolkit for Neural Machine Translation Felix Hieber Tobias Domhan Michael J. Denkowski David Vilar Artem Sokolov Ann Clifton Matt Post 44 215 0 15 Dec 2017
Gradient Descent Learns One-hidden-layer CNN: Don't be Afraid of Spurious Local Minima S. Du Jason D. Lee Yuandong Tian Barnabás Póczós Aarti Singh MLT 130 236 0 03 Dec 2017
The Implicit Bias of Gradient Descent on Separable Data Daniel Soudry Elad Hoffer Mor Shpigel Nacson Suriya Gunasekar Nathan Srebro 145 916 0 27 Oct 2017
Language Modeling with Gated Convolutional Networks Yann N. Dauphin Angela Fan Michael Auli David Grangier 212 2,391 0 23 Dec 2016
Robust Large Margin Deep Neural Networks Jure Sokolić Raja Giryes Guillermo Sapiro M. Rodrigues 64 309 0 26 May 2016
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks Tim Salimans Diederik P. Kingma ODL 186 1,940 0 25 Feb 2016