Towards Optimal Learning of Language Models

27 February 2024

Qingxiu Dong

Papers citing "Towards Optimal Learning of Language Models"

21 / 21 papers shown

Title
MiniPLM: Knowledge Distillation for Pre-Training Language Models Yuxian Gu Hao Zhou Fandong Meng Jie Zhou Minlie Huang 133 5 0 22 Oct 2024
Data Selection via Optimal Control for Language Models Yuxian Gu Li Dong Hongning Wang Y. Hao Qingxiu Dong Furu Wei Minlie Huang AI4CE 115 7 0 09 Oct 2024
Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance Jiasheng Ye Peiju Liu Tianxiang Sun Yunhua Zhou Jun Zhan Xipeng Qiu 98 75 0 25 Mar 2024
D4: Improving LLM Pretraining via Document De-Duplication and Diversification Kushal Tirumala Daniel Simig Armen Aghajanyan Ari S. Morcos SyDa 52 111 0 23 Aug 2023
LLMZip: Lossless Text Compression using Large Language Models Chandra Shekhara Kaushik Valmeekam Krishna R. Narayanan D. Kalathil J. Chamberland S. Shakkottai 53 32 0 06 Jun 2023
Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models Hong Liu Sang Michael Xie Zhiyuan Li Tengyu Ma AI4CE 108 52 0 25 Oct 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 277 2,474 0 15 Jun 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 471 6,231 0 05 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 197 1,946 0 29 Mar 2022
PPT: Pre-trained Prompt Tuning for Few-shot Learning Yuxian Gu Xu Han Zhiyuan Liu Minlie Huang VLM 90 416 0 09 Sep 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 142 849 0 14 Jun 2021
On the Origin of Implicit Regularization in Stochastic Gradient Descent Samuel L. Smith Benoit Dherin David Barrett Soham De MLT 36 203 0 28 Jan 2021
Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning Armen Aghajanyan Luke Zettlemoyer Sonal Gupta 92 562 1 22 Dec 2020
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping Minjia Zhang Yuxiong He AI4CE 42 103 0 26 Oct 2020
Estimating Training Data Influence by Tracing Gradient Descent G. Pruthi Frederick Liu Mukund Sundararajan Satyen Kale TDI 81 408 0 19 Feb 2020
On Layer Normalization in the Transformer Architecture Ruibin Xiong Yunchang Yang Di He Kai Zheng Shuxin Zheng Chen Xing Huishuai Zhang Yanyan Lan Liwei Wang Tie-Yan Liu AI4CE 128 989 0 12 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 602 4,801 0 23 Jan 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 493 42,407 0 03 Dec 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 230 996 0 01 Apr 2019
An Empirical Model of Large-Batch Training Sam McCandlish Jared Kaplan Dario Amodei OpenAI Dota Team 65 277 0 14 Dec 2018
Understanding Black-box Predictions via Influence Functions Pang Wei Koh Percy Liang TDI 203 2,885 0 14 Mar 2017