Spherical Motion Dynamics: Learning Dynamics of Neural Network with
Normalization, Weight Decay, and SGD

Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD

15 June 2020

Xiangyu Zhang

Papers citing "Spherical Motion Dynamics: Learning Dynamics of Neural Network with Normalization, Weight Decay, and SGD"

6 / 6 papers shown

Title
Batch Normalization Is Blind to the First and Second Derivatives of the Loss Zhanpeng Zhou Wen Shen Huixin Chen Ling Tang Quanshi Zhang 39 2 0 30 May 2022
The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion D. Kunin Javier Sagastuy-Breña Lauren Gillespie Eshed Margalit Hidenori Tanaka Surya Ganguli Daniel L. K. Yamins 38 16 0 19 Jul 2021
GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training Chen Zhu Renkun Ni Zheng Xu Kezhi Kong Wenjie Huang Tom Goldstein ODL 48 54 0 16 Feb 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 114 78 0 08 Dec 2020
Angle-based Search Space Shrinking for Neural Architecture Search Yiming Hu Yuding Liang Zichao Guo Ruosi Wan Xinming Zhang Yichen Wei Qingyi Gu Jian Sun 24 62 0 28 Apr 2020
On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima N. Keskar Dheevatsa Mudigere J. Nocedal M. Smelyanskiy P. T. P. Tang ODL 312 2,900 0 15 Sep 2016