The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale

25 June 2024

Papers citing "The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale"

9 / 59 papers shown

Title
Scaling Optimal LR Across Token Horizons Johan Bjorck Alon Benhaim Vishrav Chaudhary Furu Wei Xia Song 54 4 0 30 Sep 2024
Small Language Models: Survey, Measurements, and Insights Zhenyan Lu Xiang Li Dongqi Cai Rongjie Yi Fangming Liu Xiwen Zhang Nicholas D. Lane Mengwei Xu ObjD LRM 58 36 0 24 Sep 2024
Flash STU: Fast Spectral Transform Units Y. Isabel Liu Windsor Nguyen Yagiz Devre Evan Dogariu Anirudha Majumdar Elad Hazan AI4TS 72 1 0 16 Sep 2024
Improving Pretraining Data Using Perplexity Correlations Tristan Thrush Christopher Potts Tatsunori Hashimoto 34 17 0 09 Sep 2024
Masked Mixers for Language Generation and Retrieval Benjamin L. Badger 47 0 0 02 Sep 2024
Folded Context Condensation in Path Integral Formalism for Infinite Context Transformers Won-Gi Paeng Daesuk Kwon Kyungwon Jeong Honggyo Suh 71 0 0 07 May 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 239 298 0 18 Jan 2024
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 279 1,996 0 31 Dec 2020
The Woman Worked as a Babysitter: On Biases in Language Generation Emily Sheng Kai-Wei Chang Premkumar Natarajan Nanyun Peng 223 616 0 03 Sep 2019