v1v2 (latest)

Neural Networks Learn Statistics of Increasing Complexity

6 February 2024

ArXiv (abs)PDF HTML Github (34★)

Papers citing "Neural Networks Learn Statistics of Increasing Complexity"

29 / 29 papers shown

Title
A distributional simplicity bias in the learning dynamics of transformers Riccardo Rende Federica Gerace Alessandro Laio Sebastian Goldt 122 9 0 17 Feb 2025
Battle of the Backbones: A Large-Scale Comparison of Pretrained Models across Computer Vision Tasks Micah Goldblum Hossein Souri Renkun Ni Manli Shu Viraj Prabhu ... Adrien Bardes Judy Hoffman Ramalingam Chellappa Andrew Gordon Wilson Tom Goldstein VLM 159 68 0 30 Oct 2023
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling Stella Biderman Hailey Schoelkopf Quentin G. Anthony Herbie Bradley Kyle O'Brien ... USVSN Sai Prashanth Edward Raff Aviya Skowron Lintang Sutawika Oskar van der Wal 110 1,307 0 03 Apr 2023
ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders Sanghyun Woo Shoubhik Debnath Ronghang Hu Xinlei Chen Zhuang Liu In So Kweon Saining Xie SyDa 156 811 0 02 Jan 2023
Neural networks trained with SGD learn distributions of increasing complexity Maria Refinetti Alessandro Ingrosso Sebastian Goldt UQCV 120 43 0 21 Nov 2022
In-context Learning and Induction Heads Catherine Olsson Nelson Elhage Neel Nanda Nicholas Joseph Nova Dassarma ... Tom B. Brown Jack Clark Jared Kaplan Sam McCandlish C. Olah 323 528 0 24 Sep 2022
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 221 1,831 0 18 Nov 2021
The Grammar-Learning Trajectories of Neural Language Models Leshem Choshen Guy Hacohen D. Weinshall Omri Abend 97 29 0 13 Sep 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 476 2,123 0 31 Dec 2020
Deep frequency principle towards understanding why deeper learning is faster Zhi-Qin John Xu Hanxu Zhou 90 44 0 28 Jul 2020
Spectral Bias and Task-Model Alignment Explain Generalization in Kernel Regression and Infinitely Wide Neural Networks Abdulkadir Canatar Blake Bordelon Cengiz Pehlevan 120 190 0 23 Jun 2020
Designing Network Design Spaces Ilija Radosavovic Raj Prateek Kosaraju Ross B. Girshick Kaiming He Piotr Dollár GNN 107 1,697 0 30 Mar 2020
Frequency Bias in Neural Networks for Input of Non-Uniform Density Ronen Basri Meirav Galun Amnon Geifman David Jacobs Yoni Kasten S. Kritchman 92 185 0 10 Mar 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 651 4,925 0 23 Jan 2020
On the adequacy of untuned warmup for adaptive optimization Jerry Ma Denis Yarats 95 70 0 09 Oct 2019
RandAugment: Practical automated data augmentation with a reduced search space E. D. Cubuk Barret Zoph Jonathon Shlens Quoc V. Le MQ 270 3,508 0 30 Sep 2019
Understanding Generalization through Visualizations Wenjie Huang Z. Emam Micah Goldblum Liam H. Fowl J. K. Terry Furong Huang Tom Goldstein AI4CE 51 80 0 07 Jun 2019
On the Inductive Bias of Neural Tangent Kernels A. Bietti Julien Mairal 101 260 0 29 May 2019
Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks Zhi-Qin John Xu Yaoyu Zhang Yaoyu Zhang Yan Xiao Zheng Ma 131 520 0 19 Jan 2019
Reconciling modern machine learning practice and the bias-variance trade-off M. Belkin Daniel J. Hsu Siyuan Ma Soumik Mandal 249 1,660 0 28 Dec 2018
Training behavior of deep neural network in frequency domain Zhi-Qin John Xu Yaoyu Zhang Yan Xiao AI4CE 80 320 0 03 Jul 2018
On the Spectral Bias of Neural Networks Nasim Rahaman A. Baratin Devansh Arpit Felix Dräxler Min Lin Fred Hamprecht Yoshua Bengio Aaron Courville 161 1,456 0 22 Jun 2018
Neural Tangent Kernel: Convergence and Generalization in Neural Networks Arthur Jacot Franck Gabriel Clément Hongler 277 3,225 0 20 Jun 2018
Spreading vectors for similarity search Alexandre Sablayrolles Matthijs Douze Cordelia Schmid Hervé Jégou MQ 127 121 0 08 Jun 2018
Deep learning generalizes because the parameter-function map is biased towards simple functions Guillermo Valle Pérez Chico Q. Camargo A. Louis MLT AI4CE 113 232 0 22 May 2018
Deep Neural Networks as Gaussian Processes Jaehoon Lee Yasaman Bahri Roman Novak S. Schoenholz Jeffrey Pennington Jascha Narain Sohl-Dickstein UQCV BDL 139 1,100 0 01 Nov 2017
Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms Han Xiao Kashif Rasul Roland Vollgraf 285 8,928 0 25 Aug 2017
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 174 5,049 0 27 Jun 2016
On the Strong Convergence of the Optimal Linear Shrinkage Estimator for Large Dimensional Covariance Matrix Taras Bodnar Arjun K. Gupta Nestor Parolya 66 48 0 12 Aug 2013