How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model

5 July 2023

Papers citing "How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model"

24 / 24 papers shown

Title
Learning curves theory for hierarchically compositional data with power-law distributed features Francesco Cagnetta Hyunmo Kang Matthieu Wyart 100 1 0 11 May 2025
A distributional simplicity bias in the learning dynamics of transformers Riccardo Rende Federica Gerace Alessandro Laio Sebastian Goldt 107 8 0 17 Feb 2025
Probing the Latent Hierarchical Structure of Data via Diffusion Models Antonio Sclocchi Alessandro Favero Noam Itzhak Levi Matthieu Wyart DiffM 82 5 0 17 Oct 2024
Analyzing (In)Abilities of SAEs via Formal Languages Abhinav Menon Manish Shrivastava David M. Krueger Ekdeep Singh Lubana 83 8 0 15 Oct 2024
Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit Boaz Barak Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 95 132 0 18 Jul 2022
High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representation Jimmy Ba Murat A. Erdogdu Taiji Suzuki Zhichao Wang Denny Wu Greg Yang MLT 87 127 0 03 May 2022
Locality defeats the curse of dimensionality in convolutional teacher-student scenarios Alessandro Favero Francesco Cagnetta Matthieu Wyart 58 31 0 16 Jun 2021
Relative stability toward diffeomorphisms indicates performance in deep nets Leonardo Petrini Alessandro Favero Mario Geiger Matthieu Wyart OOD 50 15 0 06 May 2021
Sifting out the features by pruning: Are convolutional networks the winning lottery ticket of fully connected ones? Franco Pellegrini Giulio Biroli 86 6 0 27 Apr 2021
The Intrinsic Dimension of Images and Its Impact on Learning Phillip E. Pope Chen Zhu Ahmed Abdelkader Micah Goldblum Tom Goldstein 231 271 0 18 Apr 2021
Feature Learning in Infinite-Width Neural Networks Greg Yang J. E. Hu MLT 77 153 0 30 Nov 2020
Geometric compression of invariant manifolds in neural nets J. Paccolat Leonardo Petrini Mario Geiger Kevin Tyloo Matthieu Wyart MLT 96 36 0 22 Jul 2020
Hierarchical nucleation in deep neural networks Diego Doimo Aldo Glielmo A. Ansuini Alessandro Laio BDL AI4CE 43 31 0 07 Jul 2020
Dimensionality compression and expansion in Deep Neural Networks Stefano Recanatesi M. Farrell Madhu S. Advani Timothy Moore Guillaume Lajoie E. Shea-Brown 54 73 0 02 Jun 2019
Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation Greg Yang 146 287 0 13 Feb 2019
A mathematical theory of semantic development in deep neural networks Andrew M. Saxe James L. McClelland Surya Ganguli 73 271 0 23 Oct 2018
Nonparametric regression using deep neural networks with ReLU activation function Johannes Schmidt-Hieber 220 810 0 22 Aug 2017
Failures of Gradient-Based Deep Learning Shai Shalev-Shwartz Ohad Shamir Shaked Shammah ODL UQCV 88 201 0 23 Mar 2017
Why and When Can Deep -- but Not Shallow -- Networks Avoid the Curse of Dimensionality: a Review T. Poggio H. Mhaskar Lorenzo Rosasco Brando Miranda Q. Liao 97 576 0 02 Nov 2016
Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation Emily L. Denton Wojciech Zaremba Joan Bruna Yann LeCun Rob Fergus FAtt 177 1,689 0 02 Apr 2014
Exact solutions to the nonlinear dynamics of learning in deep linear neural networks Andrew M. Saxe James L. McClelland Surya Ganguli ODL 165 1,844 0 20 Dec 2013
Visualizing and Understanding Convolutional Networks Matthew D. Zeiler Rob Fergus FAtt SSL 595 15,876 0 12 Nov 2013
Invariant Scattering Convolution Networks Joan Bruna S. Mallat 118 1,277 0 05 Mar 2012
k-NN Regression Adapts to Local Intrinsic Dimension Samory Kpotufe 370 128 0 19 Oct 2011