Preparing Lessons for Progressive Training on Language Models

Preparing Lessons for Progressive Training on Language Models

17 January 2024

Lifeng Shang

Xin Jiang

Qun Liu

Papers citing "Preparing Lessons for Progressive Training on Language Models"

10 / 10 papers shown

Title
STEP: Staged Parameter-Efficient Pre-training for Large Language Models Kazuki Yano Takumi Ito Jun Suzuki LRM 47 1 0 05 Apr 2025
LESA: Learnable LLM Layer Scaling-Up Yifei Yang Zouying Cao Xinbei Ma Yao Yao L. Qin Z. Chen Hai Zhao 61 0 0 20 Feb 2025
Do we really have to filter out random noise in pre-training data for language models? Jinghan Ru Yuxin Xie Xianwei Zhuang Yuguo Yin Zhihui Guo Zhiming Liu Qianli Ren Yuexian Zou 83 2 0 10 Feb 2025
Measuring Social Norms of Large Language Models Ye Yuan Kexin Tang Jianhao Shen Ming Zhang Chenguang Wang ELM 34 6 0 03 Apr 2024
Measuring Vision-Language STEM Skills of Neural Models Jianhao Shen Ye Yuan Srbuhi Mirzoyan Ming Zhang Chenguang Wang VLM 33 8 0 27 Feb 2024
Speeding up Deep Model Training by Sharing Weights and Then Unsharing Shuo Yang Le Hou Xiaodan Song Qiang Liu Denny Zhou 110 9 0 08 Oct 2021
Firefly Neural Architecture Descent: a General Approach for Growing Neural Networks Lemeng Wu Bo Liu Peter Stone Qiang Liu 53 55 0 17 Feb 2021
On the Transformer Growth for Progressive BERT Training Xiaotao Gu Liyuan Liu Hongkun Yu Jing Li Cheng Chen Jiawei Han VLM 66 51 0 23 Oct 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,821 0 17 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018