Width and Depth Limits Commute in Residual Networks

1 February 2023

Papers citing "Width and Depth Limits Commute in Residual Networks"

13 / 13 papers shown

Title
The Impact of Initialization on LoRA Finetuning Dynamics Soufiane Hayou Nikhil Ghosh Bin Yu AI4CE 36 13 0 12 Jun 2024
Generalization of Scaled Deep ResNets in the Mean-Field Regime Yihang Chen Fanghui Liu Yiping Lu Grigorios G. Chrysos V. Cevher 41 2 0 14 Mar 2024
Simplifying Transformer Blocks Bobby He Thomas Hofmann 27 30 0 03 Nov 2023
Differential Equation Scaling Limits of Shaped and Unshaped Neural Networks Mufan Li Mihai Nica 28 2 0 18 Oct 2023
Wide Neural Networks as Gaussian Processes: Lessons from Deep Equilibrium Models Tianxiang Gao Xiaokai Huo Hailiang Liu Hongyang Gao BDL 25 8 0 16 Oct 2023
Tensor Programs VI: Feature Learning in Infinite-Depth Neural Networks Greg Yang Dingli Yu Chen Zhu Soufiane Hayou MLT 16 27 0 03 Oct 2023
Commutative Width and Depth Scaling in Deep Neural Networks Soufiane Hayou 49 2 0 02 Oct 2023
Depthwise Hyperparameter Transfer in Residual Networks: Dynamics and Scaling Limit Blake Bordelon Lorenzo Noci Mufan Li Boris Hanin Cengiz Pehlevan 35 22 0 28 Sep 2023
The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit Lorenzo Noci Chuning Li Mufan Li Bobby He Thomas Hofmann Chris J. Maddison Daniel M. Roy 35 31 0 30 Jun 2023
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks Blake Bordelon Cengiz Pehlevan MLT 38 29 0 06 Apr 2023
Gaussian Pre-Activations in Neural Networks: Myth or Reality? Pierre Wolinski Julyan Arbel AI4CE 76 8 0 24 May 2022
Rapid training of deep neural networks without skip connections or normalization layers using Deep Kernel Shaping James Martens Andy Ballard Guillaume Desjardins G. Swirszcz Valentin Dalibard Jascha Narain Sohl-Dickstein S. Schoenholz 88 43 0 05 Oct 2021
Stable ResNet Soufiane Hayou Eugenio Clerico Bo He George Deligiannidis Arnaud Doucet Judith Rousseau ODL SSeg 46 51 0 24 Oct 2020