Finite Versus Infinite Neural Networks: an Empirical Study

31 July 2020

Jascha Narain Sohl-Dickstein

ArXiv PDF HTML

Papers citing "Finite Versus Infinite Neural Networks: an Empirical Study"

50 / 63 papers shown

Title
The Optimization Landscape of SGD Across the Feature Learning Strength Alexander B. Atanasov Alexandru Meterez James B. Simon Cengiz Pehlevan 43 2 0 06 Oct 2024
Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective Jingren Liu Zhong Ji YunLong Yu Jiale Cao Yanwei Pang Jungong Han Xuelong Li CLL 42 5 0 24 Jul 2024
Equivariant Neural Tangent Kernels Philipp Misof Pan Kessel Jan E. Gerken 64 0 0 10 Jun 2024
NTK-Guided Few-Shot Class Incremental Learning Jingren Liu Zhong Ji Yanwei Pang YunLong Yu CLL 39 3 0 19 Mar 2024
Weak Correlations as the Underlying Principle for Linearization of Gradient-Based Learning Systems Ori Shem-Ur Yaron Oz 19 0 0 08 Jan 2024
On the Neural Tangent Kernel of Equilibrium Models Zhili Feng J. Zico Kolter 18 6 0 21 Oct 2023
Connecting NTK and NNGP: A Unified Theoretical Framework for Wide Neural Network Learning Dynamics Yehonatan Avidan Qianyi Li H. Sompolinsky 60 8 0 08 Sep 2023
Sparsity-depth Tradeoff in Infinitely Wide Deep Neural Networks Chanwoo Chun Daniel D. Lee BDL 35 2 0 17 May 2023
Provable Guarantees for Nonlinear Feature Learning in Three-Layer Neural Networks Eshaan Nichani Alexandru Damian Jason D. Lee MLT 44 13 0 11 May 2023
Do deep neural networks have an inbuilt Occam's razor? Chris Mingard Henry Rees Guillermo Valle Pérez A. Louis UQCV BDL 21 16 0 13 Apr 2023
Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks Blake Bordelon Cengiz Pehlevan MLT 38 29 0 06 Apr 2023
Wide neural networks: From non-gaussian random fields at initialization to the NTK geometry of training Luís Carvalho Joao L. Costa José Mourao Gonccalo Oliveira AI4CE 26 1 0 06 Apr 2023
Dataset Distillation with Convexified Implicit Gradients Noel Loo Ramin Hasani Mathias Lechner Daniela Rus DD 31 41 0 13 Feb 2023
On a continuous time model of gradient descent dynamics and instability in deep learning Mihaela Rosca Yan Wu Chongli Qin Benoit Dherin 18 6 0 03 Feb 2023
Bayes-optimal Learning of Deep Random Networks of Extensive-width Hugo Cui Florent Krzakala Lenka Zdeborová BDL 20 35 0 01 Feb 2023
A Simple Algorithm For Scaling Up Kernel Methods Tengyu Xu Bryan Kelly Semyon Malamud 16 0 0 26 Jan 2023
Dataset Distillation: A Comprehensive Review Ruonan Yu Songhua Liu Xinchao Wang DD 53 121 0 17 Jan 2023
Data Distillation: A Survey Noveen Sachdeva Julian McAuley DD 45 73 0 11 Jan 2023
Characterizing the Spectrum of the NTK via a Power Series Expansion Michael Murray Hui Jin Benjamin Bowman Guido Montúfar 38 11 0 15 Nov 2022
A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks Zhengdao Chen Eric Vanden-Eijnden Joan Bruna MLT 27 5 0 28 Oct 2022
Evolution of Neural Tangent Kernels under Benign and Adversarial Training Noel Loo Ramin Hasani Alexander Amini Daniela Rus AAML 34 13 0 21 Oct 2022
Scale-invariant Bayesian Neural Networks with Connectivity Tangent Kernel Sungyub Kim Si-hun Park Kyungsu Kim Eunho Yang BDL 29 4 0 30 Sep 2022
Joint Embedding Self-Supervised Learning in the Kernel Regime B. Kiani Randall Balestriero Yubei Chen S. Lloyd Yann LeCun SSL 49 13 0 29 Sep 2022
Approximation results for Gradient Descent trained Shallow Neural Networks in $1d$ R. Gentile G. Welper ODL 52 6 0 17 Sep 2022
Open Source Vizier: Distributed Infrastructure and API for Reliable and Flexible Blackbox Optimization Xingyou Song Sagi Perel Chansoo Lee Greg Kochanski Daniel Golovin 31 26 0 27 Jul 2022
Learning sparse features can lead to overfitting in neural networks Leonardo Petrini Francesco Cagnetta Eric Vanden-Eijnden M. Wyart MLT 42 23 0 24 Jun 2022
Fast Finite Width Neural Tangent Kernel Roman Novak Jascha Narain Sohl-Dickstein S. Schoenholz AAML 22 53 0 17 Jun 2022
Identifying good directions to escape the NTK regime and efficiently learn low-degree plus sparse polynomials Eshaan Nichani Yunzhi Bai Jason D. Lee 29 10 0 08 Jun 2022
Infinite Recommendation Networks: A Data-Centric Approach Noveen Sachdeva Mehak Preet Dhaliwal Carole-Jean Wu Julian McAuley DD 33 28 0 03 Jun 2022
On Feature Learning in Neural Networks with Global Convergence Guarantees Zhengdao Chen Eric Vanden-Eijnden Joan Bruna MLT 36 13 0 22 Apr 2022
Contrasting random and learned features in deep Bayesian linear regression Jacob A. Zavatone-Veth William L. Tong Cengiz Pehlevan BDL MLT 28 26 0 01 Mar 2022
Explicitising The Implicit Intrepretability of Deep Neural Networks Via Duality Chandrashekar Lakshminarayanan Ashutosh Kumar Singh A. Rajkumar AI4CE 26 1 0 01 Mar 2022
Investigating Power laws in Deep Representation Learning Arna Ghosh Arnab Kumar Mondal Kumar Krishna Agrawal Blake A. Richards SSL OOD 14 19 0 11 Feb 2022
Tight Convergence Rate Bounds for Optimization Under Power Law Spectral Conditions Maksim Velikanov Dmitry Yarotsky 9 6 0 02 Feb 2022
A generalization gap estimation for overparameterized models via the Langevin functional variance Akifumi Okuno Keisuke Yano 38 1 0 07 Dec 2021
Depth induces scale-averaging in overparameterized linear Bayesian neural networks Jacob A. Zavatone-Veth Cengiz Pehlevan BDL UQCV MDE 41 8 0 23 Nov 2021
Subquadratic Overparameterization for Shallow Neural Networks Chaehwan Song Ali Ramezani-Kebrya Thomas Pethick Armin Eftekhari V. Cevher 30 31 0 02 Nov 2021
Neural Networks as Kernel Learners: The Silent Alignment Effect Alexander B. Atanasov Blake Bordelon Cengiz Pehlevan MLT 26 75 0 29 Oct 2021
Deep Active Learning by Leveraging Training Dynamics Haonan Wang Wei Huang Ziwei Wu A. Margenot Hanghang Tong Jingrui He AI4CE 27 33 0 16 Oct 2021
Bayesian neural network unit priors and generalized Weibull-tail property M. Vladimirova Julyan Arbel Stéphane Girard BDL 54 9 0 06 Oct 2021
Simple, Fast, and Flexible Framework for Matrix Completion with Infinite Width Neural Networks Adityanarayanan Radhakrishnan George Stefanakis M. Belkin Caroline Uhler 30 25 0 31 Jul 2021
Dataset Distillation with Infinitely Wide Convolutional Networks Timothy Nguyen Roman Novak Lechao Xiao Jaehoon Lee DD 51 229 0 27 Jul 2021
How to Train Your Wide Neural Network Without Backprop: An Input-Weight Alignment Perspective Akhilan Boopathy Ila Fiete 24 9 0 15 Jun 2021
The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective Geoff Pleiss John P. Cunningham 28 24 0 11 Jun 2021
A Neural Tangent Kernel Perspective of GANs Jean-Yves Franceschi Emmanuel de Bézenac Ibrahim Ayed Mickaël Chen Sylvain Lamprier Patrick Gallinari 34 26 0 10 Jun 2021
A self consistent theory of Gaussian Processes captures feature learning effects in finite CNNs Gadi Naveh Zohar Ringel SSL MLT 36 31 0 08 Jun 2021
Relative stability toward diffeomorphisms indicates performance in deep nets Leonardo Petrini Alessandro Favero Mario Geiger M. Wyart OOD 38 15 0 06 May 2021
A Neural Pre-Conditioning Active Learning Algorithm to Reduce Label Complexity Seo Taek Kong Soomin Jeon Dongbin Na Jaewon Lee Honglak Lee Kyu-Hwan Jung 23 6 0 08 Apr 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 27 250 0 12 Feb 2021
A Convergence Theory Towards Practical Over-parameterized Deep Neural Networks Asaf Noy Yi Tian Xu Y. Aflalo Lihi Zelnik-Manor R. L. Jin 36 3 0 12 Jan 2021