Title
Don't be lazy: CompleteP enables compute-efficient deep transformers Nolan Dey Bin Claire Zhang Lorenzo Noci Mufan Bill Li Blake Bordelon Shane Bergsma C. Pehlevan Boris Hanin Joel Hestness 41 0 0 02 May 2025
AeroLite: Tag-Guided Lightweight Generation of Aerial Image Captions Xing Zi Tengjun Ni Xianjing Fan Xian Tao Jun Li Ali Braytee Mukesh Prasad 23 0 0 13 Apr 2025
Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions E. Liu Amanda Bertsch Lintang Sutawika Lindia Tjuatja Patrick Fernandes ... Shri Kiran Srinivasan Carolin (Haas) Lawrence Aditi Raghunathan Kiril Gashteovski Graham Neubig 90 0 0 05 Mar 2025
Crystal: Illuminating LLM Abilities on Language and Code Tianhua Tao Junbo Li Bowen Tan Hongyi Wang William Marshall ... Joel Hestness Natalia Vassilieva Zhiqiang Shen Eric P. Xing Zhengzhong Liu 47 4 0 06 Nov 2024
How Does Critical Batch Size Scale in Pre-training? Hanlin Zhang Depen Morwani Nikhil Vyas Jingfeng Wu Difan Zou Udaya Ghai Dean Phillips Foster Sham Kakade 77 8 0 29 Oct 2024
$u-$\mu$P: The Unit-Scaled Maximal Update Parametrization$ u- $\mu$ P: The Unit-Scaled Maximal Update Parametrization Charlie Blake C. Eichenberg Josef Dean Lukas Balles Luke Y. Prince Bjorn Deiseroth Andres Felipe Cruz Salinas Carlo Luschi Samuel Weinbach Douglas Orr 55 9 0 24 Jul 2024
Sparse maximal update parameterization: A holistic approach to sparse training dynamics Nolan Dey Shane Bergsma Joel Hestness 38 5 0 24 May 2024
Does Transformer Interpretability Transfer to RNNs? Gonccalo Paulo Thomas Marshall Nora Belrose 57 6 0 09 Apr 2024
MosaicBERT: A Bidirectional Encoder Optimized for Fast Pretraining Jacob P. Portes Alex Trott Sam Havens Daniel King Abhinav Venigalla Moin Nadeem Nikhil Sardana D. Khudia Jonathan Frankle 23 16 0 29 Dec 2023
I Know You Did Not Write That! A Sampling Based Watermarking Method for Identifying Machine Generated Text Kaan Efe Keles Ömer Kaan Gürbüz Mucahid Kutlu WaLM 16 1 0 29 Nov 2023
Towards the Law of Capacity Gap in Distilling Language Models Chen Zhang Dawei Song Zheyu Ye Yan Gao ELM 35 20 0 13 Nov 2023
Position Interpolation Improves ALiBi Extrapolation Faisal Al-Khateeb Nolan Dey Daria Soboleva Joel Hestness LLMSV 21 5 0 18 Oct 2023
What Language Model to Train if You Have One Million GPU Hours? Teven Le Scao Thomas Wang Daniel Hesslow Lucile Saulnier Stas Bekman ... Lintang Sutawika Jaesung Tae Zheng-Xin Yong Julien Launay Iz Beltagy MoE AI4CE 230 103 0 27 Oct 2022
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press Noah A. Smith M. Lewis 253 695 0 27 Aug 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 261 1,996 0 31 Dec 2020
The Woman Worked as a Babysitter: On Biases in Language Generation Emily Sheng Kai-Wei Chang Premkumar Natarajan Nanyun Peng 223 616 0 03 Sep 2019