AutoInit: Automatic Initialization via Jacobian Tuning

27 June 2022

Papers citing "AutoInit: Automatic Initialization via Jacobian Tuning"

6 / 6 papers shown

Title
Geometric Dynamics of Signal Propagation Predict Trainability of Transformers Aditya Cowsik Tamra M. Nebabu Xiao-Liang Qi Surya Ganguli 25 9 0 05 Mar 2024
Gradient Flossing: Improving Gradient Descent through Dynamic Control of Jacobians Rainer Engelken 21 5 0 28 Dec 2023
Feature Learning and Generalization in Deep Networks with Orthogonal Weights Hannah Day Yonatan Kahn Daniel A. Roberts OOD 24 1 0 11 Oct 2023
Critical Initialization of Wide and Deep Neural Networks through Partial Jacobians: General Theory and Applications Darshil Doshi Tianyu He Andrey Gromov 27 8 0 23 Nov 2021
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping James Martens Andy Ballard Guillaume Desjardins G. Swirszcz Valentin Dalibard Jascha Narain Sohl-Dickstein S. Schoenholz 88 43 0 05 Oct 2021
Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networks Lechao Xiao Yasaman Bahri Jascha Narain Sohl-Dickstein S. Schoenholz Jeffrey Pennington 227 348 0 14 Jun 2018