gzip Predicts Data-dependent Scaling Laws

26 May 2024

Papers citing "gzip Predicts Data-dependent Scaling Laws"

12 / 12 papers shown

Title
RegMix: Data Mixture as Regression for Language Model Pre-training Qian Liu Xiaosen Zheng Niklas Muennighoff Guangtao Zeng Longxu Dou Tianyu Pang Jing Jiang Min Lin MoE 156 52 1 01 Jul 2024
Mission: Impossible Language Models Julie Kallini Isabel Papadimitriou Richard Futrell Kyle Mahowald Christopher Potts ELM LRM 88 21 0 12 Jan 2024
Scaling Laws for Generative Mixed-Modal Language Models Armen Aghajanyan L. Yu Alexis Conneau Wei-Ning Hsu Karen Hambardzumyan Susan Zhang Stephen Roller Naman Goyal Omer Levy Luke Zettlemoyer MoE VLM 78 108 0 10 Jan 2023
Beyond neural scaling laws: beating power law scaling via data pruning Ben Sorscher Robert Geirhos Shashank Shekhar Surya Ganguli Ari S. Morcos 97 444 0 29 Jun 2022
Estimating the Entropy of Linguistic Distributions Aryaman Arora Clara Meister Ryan Cotterell 37 11 0 04 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,976 0 29 Mar 2022
Scaling Scaling Laws with Board Games Andrew Jones 50 43 0 07 Apr 2021
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 608 4,893 0 23 Jan 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 529 42,559 0 03 Dec 2019
A Constructive Prediction of the Generalization Error Across Scales Jonathan S. Rosenfeld Amir Rosenfeld Yonatan Belinkov Nir Shavit 105 215 0 27 Sep 2019
Deep Learning Scaling is Predictable, Empirically Joel Hestness Sharan Narang Newsha Ardalani G. Diamos Heewoo Jun Hassan Kianinejad Md. Mostofa Ali Patwary Yang Yang Yanqi Zhou 107 742 0 01 Dec 2017
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 344 8,169 0 13 Aug 2016