(Mis)Fitting: A Survey of Scaling Laws

26 February 2025

Papers citing "(Mis)Fitting: A Survey of Scaling Laws"

41 / 41 papers shown

Title
LCDB 1.1: A Database Illustrating Learning Curves Are More Ill-Behaved Than Previously Thought Cheng Yan Felix Mohr Tom Viering 88 0 0 21 May 2025
The Sparse Frontier: Sparse Attention Trade-offs in Transformer LLMs Piotr Nawrot Robert Li Renjie Huang Sebastian Ruder Kelly Marchisio Edoardo Ponti 91 5 0 24 Apr 2025
RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs Zhongzhan Huang Guoming Ling Vincent S. Liang Yupei Lin Yandong Chen Shanshan Zhong Hefeng Wu LRM 200 7 0 08 Mar 2025
Resolving Discrepancies in Compute-Optimal Scaling of Language Models Tomer Porian Mitchell Wortsman J. Jitsev Ludwig Schmidt Y. Carmon 162 26 0 27 Jun 2024
Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations Alexander Hägele Elie Bakouch Atli Kosson Loubna Ben Allal Leandro von Werra Martin Jaggi 91 45 0 28 May 2024
Chinchilla Scaling: A replication attempt T. Besiroglu Ege Erdil Matthew Barnett Josh You 92 24 0 15 Apr 2024
Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic Sachin Goyal Pratyush Maini Zachary Chase Lipton Aditi Raghunathan J. Zico Kolter 100 46 0 10 Apr 2024
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws Nikhil Sardana Jacob P. Portes Sasha Doubov Jonathan Frankle LRM 355 88 0 31 Dec 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.5K 14,761 0 15 Mar 2023
Scaling laws for single-agent reinforcement learning Jacob Hilton Jie Tang John Schulman 111 22 0 31 Jan 2023
Scaling Laws for Generative Mixed-Modal Language Models Armen Aghajanyan L. Yu Alexis Conneau Wei-Ning Hsu Karen Hambardzumyan Susan Zhang Stephen Roller Naman Goyal Omer Levy Luke Zettlemoyer MoE VLM 86 110 0 10 Jan 2023
Scaling Laws for a Multi-Agent Reinforcement Learning Model Oren Neumann C. Gros 86 27 0 29 Sep 2022
Revisiting Neural Scaling Laws in Language and Vision Ibrahim Alabdulmohsin Behnam Neyshabur Xiaohua Zhai 220 111 0 13 Sep 2022
Understanding Scaling Laws for Recommendation Models Newsha Ardalani Carole-Jean Wu Zeliang Chen Bhargav Bhushanam Adnan Aziz 90 31 0 17 Aug 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 119 664 0 15 Aug 2022
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung W. Fedus J. Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler AI4CE 111 106 0 21 Jul 2022
Beyond neural scaling laws: beating power law scaling via data pruning Ben Sorscher Robert Geirhos Shashank Shekhar Surya Ganguli Ari S. Morcos 100 444 0 29 Jun 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 292 2,521 0 15 Jun 2022
Scaling Laws and Interpretability of Learning from Repeated Data Danny Hernandez Tom B. Brown Tom Conerly Nova Dassarma Dawn Drain ... Catherine Olsson Dario Amodei Nicholas Joseph Jared Kaplan Sam McCandlish 77 118 0 21 May 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer Greg Yang J. E. Hu Igor Babuschkin Szymon Sidor Xiaodong Liu David Farhi Nick Ryder J. Pachocki Weizhu Chen Jianfeng Gao 114 168 0 07 Mar 2022
Scaling Laws Under the Microscope: Predicting Transformer Performance from Small Scale Experiments Maor Ivgi Y. Carmon Jonathan Berant 72 17 0 13 Feb 2022
Examining Scaling and Transfer of Language Model Architectures for Machine Translation Biao Zhang Behrooz Ghorbani Ankur Bapna Yong Cheng Xavier Garcia Jonathan Shen Orhan Firat 73 23 0 01 Feb 2022
Scaling Law for Recommendation Models: Towards General-purpose User Representations Kyuyong Shin Hanock Kwak KyungHyun Kim Max Nihlén Ramström Jisu Jeong Jung-Woo Ha Seon Gyeom Kim ELM 105 42 0 15 Nov 2021
Scaling Laws for the Few-Shot Adaptation of Pre-trained Image Classifiers Gabriele Prato Simon Guiroy Ethan Caballero Irina Rish Sarath Chandar VLM 84 12 0 13 Oct 2021
Scaling Laws for Neural Machine Translation Behrooz Ghorbani Orhan Firat Markus Freitag Ankur Bapna M. Krikun Xavier Garcia Ciprian Chelba Colin Cherry 85 102 0 16 Sep 2021
A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective? Hiroaki Mikami Kenji Fukumizu Shogo Murai Shuji Suzuki Yuta Kikuchi Taiji Suzuki S. Maeda Kohei Hayashi 78 12 0 25 Aug 2021
Scaling Laws for Acoustic Models J. Droppo Oguz H. Elibol 60 23 0 11 Jun 2021
RoFormer: Enhanced Transformer with Rotary Position Embedding Jianlin Su Yu Lu Shengfeng Pan Ahmed Murtadha Bo Wen Yunfeng Liu 329 2,533 0 20 Apr 2021
Scaling Scaling Laws with Board Games Andrew Jones 55 43 0 07 Apr 2021
Explaining Neural Scaling Laws Yasaman Bahri Ethan Dyer Jared Kaplan Jaehoon Lee Utkarsh Sharma 78 269 0 12 Feb 2021
Scaling Laws for Transfer Danny Hernandez Jared Kaplan T. Henighan Sam McCandlish 95 251 0 02 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 476 2,123 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 651 4,925 0 23 Jan 2020
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 506 20,376 0 23 Oct 2019
A Constructive Prediction of the Generalization Error Across Scales Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit 105 215 0 27 Sep 2019
An Empirical Model of Large-Batch Training Sam McCandlish Jared Kaplan Dario Amodei OpenAI Dota Team 74 280 0 14 Dec 2018
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 209 3,534 0 19 Aug 2018
Deep Learning Scaling is Predictable, Empirically Joel Hestness Sharan Narang Newsha Ardalani G. Diamos Heewoo Jun Hassan Kianinejad Md. Mostofa Ali Patwary Yang Yang Yanqi Zhou 112 744 0 01 Dec 2017
To prune, or not to prune: exploring the efficacy of pruning for model compression Michael Zhu Suyog Gupta 202 1,282 0 05 Oct 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 808 132,725 0 12 Jun 2017