Learning Curve Theory

8 February 2021

Marcus Hutter

Papers citing "Learning Curve Theory"

47 / 47 papers shown

Title
Scaling Laws and Representation Learning in Simple Hierarchical Languages: Transformers vs. Convolutional Architectures Francesco Cagnetta Alessandro Favero Antonio Sclocchi M. Wyart 26 0 0 11 May 2025
Learning curves theory for hierarchically compositional data with power-law distributed features Francesco Cagnetta Hyunmo Kang M. Wyart 36 0 0 11 May 2025
Quiet Feature Learning in Algorithmic Tasks Prudhviraj Naidu Zixian Wang Leon Bergen R. Paturi VLM 54 0 0 06 May 2025
A Multi-Power Law for Loss Curve Prediction Across Learning Rate Schedules Kairong Luo Haodong Wen Shengding Hu Zhenbo Sun Zhiyuan Liu Maosong Sun Kaifeng Lyu Wenguang Chen CLL 59 1 0 17 Mar 2025
Position: Solve Layerwise Linear Models First to Understand Neural Dynamical Phenomena (Neural Collapse, Emergence, Lazy/Rich Regime, and Grokking) Yoonsoo Nam Seok Hyeong Lee Clementine Domine Yea Chan Park Charles London Wonyl Choi Niclas Goring Seungjai Lee AI4CE 38 0 0 28 Feb 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 108 3 0 24 Feb 2025
How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines Ayan Sengupta Yash Goel Tanmoy Chakraborty 50 0 0 17 Feb 2025
Loss-to-Loss Prediction: Scaling Laws for All Datasets David Brandfonbrener Nikhil Anand Nikhil Vyas Eran Malach Sham Kakade 77 3 0 19 Nov 2024
Scaling Laws for Pre-training Agents and World Models Tim Pearce Tabish Rashid Dave Bignell Raluca Georgescu Sam Devlin Katja Hofmann LM&Ro 40 6 0 07 Nov 2024
A Simple Model of Inference Scaling Laws Noam Levi LRM 32 6 0 21 Oct 2024
Adaptive Data Optimization: Dynamic Sample Selection with Scaling Laws Yiding Jiang Allan Zhou Zhili Feng Sadhika Malladi J. Zico Kolter 39 15 0 15 Oct 2024
Analyzing Neural Scaling Laws in Two-Layer Networks with Power-Law Data Spectra Roman Worschech B. Rosenow 41 0 0 11 Oct 2024
Unified Neural Network Scaling Laws and Scale-time Equivalence Akhilan Boopathy Ila Fiete 40 0 0 09 Sep 2024
Breaking Neural Network Scaling Laws with Modularity Akhilan Boopathy Sunshine Jiang William Yue Jaedong Hwang Abhiram Iyer Ila Fiete OOD 39 2 0 09 Sep 2024
Towards Exact Computation of Inductive Bias Akhilan Boopathy William Yue Jaedong Hwang Abhiram Iyer Ila Fiete 34 0 0 22 Jun 2024
Scaling Laws in Linear Regression: Compute, Parameters, and Data Licong Lin Jingfeng Wu Sham Kakade Peter L. Bartlett Jason D. Lee LRM 41 15 0 12 Jun 2024
Reconciling Kaplan and Chinchilla Scaling Laws Tim Pearce Jinyeop Song 34 8 0 12 Jun 2024
Scaling Laws for the Value of Individual Data Points in Machine Learning Ian Covert Wenlong Ji Tatsunori Hashimoto James Y. Zou TDI 37 8 0 30 May 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 46 62 0 25 Mar 2024
How much data do you need? Part 2: Predicting DL class specific training dataset sizes Thomas Mühlenstädt Jelena Frtunikj 19 2 0 10 Mar 2024
A Tale of Tails: Model Collapse as a Change of Scaling Laws Elvis Dohmatob Yunzhen Feng Pu Yang Francois Charton Julia Kempe 21 62 0 10 Feb 2024
Scaling Laws for Associative Memories Vivien A. Cabannes Elvis Dohmatob A. Bietti 16 19 0 04 Oct 2023
Pareto Frontiers in Neural Feature Learning: Data, Compute, Width, and Luck Benjamin L. Edelman Surbhi Goel Sham Kakade Eran Malach Cyril Zhang 48 8 0 07 Sep 2023
Is One Epoch All You Need For Multi-Fidelity Hyperparameter Optimization? Romain Egele Isabelle M Guyon Yixuan Sun Prasanna Balaprakash 32 2 0 28 Jul 2023
The semantic landscape paradigm for neural networks Shreyas Gokhale 21 2 0 18 Jul 2023
Delegated Classification Eden Saig Inbal Talgam-Cohen Nir Rosenfeld 16 7 0 20 Jun 2023
Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design Ibrahim M. Alabdulmohsin Xiaohua Zhai Alexander Kolesnikov Lucas Beyer VLM 27 57 0 22 May 2023
Model-agnostic Measure of Generalization Difficulty Akhilan Boopathy Kevin Liu Jaedong Hwang Shu Ge Asaad Mohammedsaleh Ila Fiete 72 4 0 01 May 2023
The Quantization Model of Neural Scaling Eric J. Michaud Ziming Liu Uzay Girit Max Tegmark MILM 27 77 0 23 Mar 2023
Scaling Laws for Multilingual Neural Machine Translation Patrick Fernandes Behrooz Ghorbani Xavier Garcia Markus Freitag Orhan Firat 30 29 0 19 Feb 2023
A Simplistic Model of Neural Scaling Laws: Multiperiodic Santa Fe Processes L. Debowski MILM 19 11 0 17 Feb 2023
Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer Benjamin Muller Deepanshu Gupta Siddharth Patwardhan J. Fauconnier David Vandyke Sachin Agarwal 33 5 0 04 Dec 2022
A Solvable Model of Neural Scaling Laws A. Maloney Daniel A. Roberts J. Sully 36 51 0 30 Oct 2022
Revisiting Neural Scaling Laws in Language and Vision Ibrahim M. Alabdulmohsin Behnam Neyshabur Xiaohua Zhai 156 102 0 13 Sep 2022
How Much More Data Do I Need? Estimating Requirements for Downstream Tasks Rafid Mahmood James Lucas David Acuna Daiqing Li Jonah Philion Jose M. Alvarez Zhiding Yu Sanja Fidler M. Law 14 26 0 04 Jul 2022
Unified Scaling Laws for Routed Language Models Aidan Clark Diego de Las Casas Aurelia Guy A. Mensch Michela Paganini ... Oriol Vinyals Jack W. Rae Erich Elsen Koray Kavukcuoglu Karen Simonyan MoE 27 177 0 02 Feb 2022
Error Scaling Laws for Kernel Classification under Source and Capacity Conditions Hugo Cui Bruno Loureiro Florent Krzakala Lenka Zdeborová 46 10 0 29 Jan 2022
Scaling Law for Recommendation Models: Towards General-purpose User Representations Kyuyong Shin Hanock Kwak KyungHyun Kim Max Nihlén Ramström Jisu Jeong Jung-Woo Ha S. Kim ELM 36 38 0 15 Nov 2021
Turing-Universal Learners with Optimal Scaling Laws Preetum Nakkiran 21 2 0 09 Nov 2021
Scaling Laws for Neural Machine Translation Behrooz Ghorbani Orhan Firat Markus Freitag Ankur Bapna M. Krikun Xavier Garcia Ciprian Chelba Colin Cherry 32 99 0 16 Sep 2021
A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective? Hiroaki Mikami Kenji Fukumizu Shogo Murai Shuji Suzuki Yuta Kikuchi Taiji Suzuki S. Maeda Kohei Hayashi 40 12 0 25 Aug 2021
Scaling Scaling Laws with Board Games Andrew Jones 8 38 0 07 Apr 2021
The Shape of Learning Curves: a Review T. Viering Marco Loog 18 122 0 19 Mar 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 27 250 0 12 Feb 2021
Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks Blake Bordelon Abdulkadir Canatar C. Pehlevan 144 201 0 07 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 237 4,469 0 23 Jan 2020
Dataset Growth in Medical Image Analysis Research Y. Landau N. Kiryati OOD 13 30 0 21 Aug 2019