Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics

4 August 2020

Papers citing "Shallow Univariate ReLu Networks as Splines: Initialization, Loss Surface, Hessian, & Gradient Flow Dynamics"

37 / 37 papers shown

Title
On the Geometry of Deep Learning Randall Balestriero Ahmed Imtiaz Humayun Richard G. Baraniuk AI4CE 104 1 0 09 Aug 2024
Equidistribution-based training of Free Knot Splines and ReLU Neural Networks Simone Appella S. Arridge Chris Budd Teo Deveney L. Kreusser 65 0 0 02 Jul 2024
Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embedding Zhengqing Wu Berfin Simsek Francois Ged ODL 84 0 0 08 Feb 2024
Symmetry-aware Neural Architecture for Embodied Visual Navigation Shuang Liu Takayuki Okatani 51 1 0 17 Dec 2021
Emergence of Lie symmetries in functional architectures learned by CNNs F. Bertoni Noemi Montobbio A. Sarti G. Citti 31 5 0 17 Apr 2021
Symmetry-Aware Reservoir Computing W. A. S. Barbosa Aaron Griffith G. Rowlands L. Govia G. Ribeill M. Nguyen T. Ohki D. Gauthier 36 12 0 30 Jan 2021
Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics D. Kunin Javier Sagastuy-Breña Surya Ganguli Daniel L. K. Yamins Hidenori Tanaka 144 80 0 08 Dec 2020
A Deeper Look at the Hessian Eigenspectrum of Deep Neural Networks and its Applications to Regularization Adepu Ravi Sankar Yash Khasbage Rahul Vigneswaran V. Balasubramanian 77 44 0 07 Dec 2020
Inverse Problems, Deep Learning, and Symmetry Breaking Kshitij Tayal Chieh-Hsin Lai Vipin Kumar Ju Sun AI4CE 88 15 0 20 Mar 2020
Reverse-Engineering Deep ReLU Networks David Rolnick Konrad Paul Kording 74 104 0 02 Oct 2019
How noise affects the Hessian spectrum in overparameterized neural networks Ming-Bo Wei D. Schwab 65 28 0 01 Oct 2019
Gradient Dynamics of Shallow Univariate ReLU Networks Francis Williams Matthew Trager Claudio Silva Daniele Panozzo Denis Zorin Joan Bruna 62 80 0 18 Jun 2019
Kernel and Rich Regimes in Overparametrized Models Blake E. Woodworth Suriya Gunasekar Pedro H. P. Savarese E. Moroshko Itay Golan Jason D. Lee Daniel Soudry Nathan Srebro 80 364 0 13 Jun 2019
Deep ReLU Networks Have Surprisingly Few Activation Patterns Boris Hanin David Rolnick 84 227 0 03 Jun 2019
On Exact Computation with an Infinitely Wide Neural Net Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruslan Salakhutdinov Ruosong Wang 226 925 0 26 Apr 2019
A Sober Look at Neural Network Initializations Ingo Steinwart 36 9 0 27 Mar 2019
How do infinite width bounded norm networks look in function space? Pedro H. P. Savarese Itay Evron Daniel Soudry Nathan Srebro 80 166 0 13 Feb 2019
An Investigation into Neural Net Optimization via Hessian Eigenvalue Density Behrooz Ghorbani Shankar Krishnan Ying Xiao ODL 67 324 0 29 Jan 2019
Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networks Sanjeev Arora S. Du Wei Hu Zhiyuan Li Ruosong Wang MLT 202 972 0 24 Jan 2019
Scaling description of generalization with number of parameters in deep learning Mario Geiger Arthur Jacot S. Spigler Franck Gabriel Levent Sagun Stéphane dÁscoli Giulio Biroli Clément Hongler Matthieu Wyart 83 196 0 06 Jan 2019
On Lazy Training in Differentiable Programming Lénaïc Chizat Edouard Oyallon Francis R. Bach 111 835 0 19 Dec 2018
Spline Regression with Automatic Knot Selection Vivien Goepp Olivier Bouaziz G. Nuel 25 31 0 06 Aug 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 269 3,203 0 20 Jun 2018
A Mean Field View of the Landscape of Two-Layers Neural Networks Song Mei Andrea Montanari Phan-Minh Nguyen MLT 98 861 0 18 Apr 2018
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks Jonathan Frankle Michael Carbin 240 3,473 0 09 Mar 2018
Visualizing the Loss Landscape of Neural Nets Hao Li Zheng Xu Gavin Taylor Christoph Studer Tom Goldstein 252 1,893 0 28 Dec 2017
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 131 1,093 0 01 Nov 2017
High-dimensional dynamics of generalization error in neural networks Madhu S. Advani Andrew M. Saxe AI4CE 139 469 0 10 Oct 2017
Towards Understanding Generalization of Deep Learning: Perspective of Loss Landscapes Lei Wu Zhanxing Zhu E. Weinan ODL 64 221 0 30 Jun 2017
Empirical Analysis of the Hessian of Over-Parametrized Neural Networks Levent Sagun Utku Evci V. U. Güney Yann N. Dauphin Léon Bottou 54 418 0 14 Jun 2017
The loss surface of deep and wide neural networks Quynh N. Nguyen Matthias Hein ODL 164 284 0 26 Apr 2017
Understanding deep learning requires rethinking generalization Chiyuan Zhang Samy Bengio Moritz Hardt Benjamin Recht Oriol Vinyals HAI 342 4,629 0 10 Nov 2016
Fast Algorithms for Segmented Regression Jayadev Acharya Ilias Diakonikolas Jerry Li Ludwig Schmidt 57 31 0 14 Jul 2016
Symmetry-aware Depth Estimation using Deep Neural Networks Guilin Liu Chao Yang Zimo Li Duygu Ceylan Qi-Xing Huang MDE 47 3 0 20 Apr 2016
Symmetry-invariant optimization in deep networks Vijay Badrinarayanan Bamdev Mishra R. Cipolla ODL 48 34 0 05 Nov 2015
Path-SGD: Path-Normalized Optimization in Deep Neural Networks Behnam Neyshabur Ruslan Salakhutdinov Nathan Srebro ODL 86 309 0 08 Jun 2015
Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification Kaiming He Xinming Zhang Shaoqing Ren Jian Sun VLM 326 18,625 0 06 Feb 2015