EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets

31 December 2020

Papers citing "EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets"

25 / 25 papers shown

Title
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training Yuezhou Hu Jun-Jie Zhu Jianfei Chen 45 0 0 13 Sep 2024
Light-PEFT: Lightening Parameter-Efficient Fine-Tuning via Early Pruning Naibin Gu Peng Fu Xiyu Liu Bowen Shen Zheng-Shen Lin Weiping Wang 38 6 0 06 Jun 2024
Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training Hyesong Choi Hyejin Park Kwang Moo Yi Sungmin Cha Dongbo Min 39 9 0 12 Apr 2024
NeuroPrune: A Neuro-inspired Topological Sparse Training Algorithm for Large Language Models Amit Dhurandhar Tejaswini Pedapati Ronny Luss Soham Dan Aurélie C. Lozano Payel Das Georgios Kollias 22 3 0 28 Feb 2024
The Snowflake Hypothesis: Training Deep GNN with One Node One Receptive field Kun Wang Guohao Li Shilong Wang Guibin Zhang Kaidi Wang Yang You Xiaojiang Peng Keli Zhang Yang Wang 37 8 0 19 Aug 2023
Gradient-Free Structured Pruning with Unlabeled Data Azade Nova H. Dai Dale Schuurmans SyDa 40 20 0 07 Mar 2023
A Survey on Efficient Training of Transformers Bohan Zhuang Jing Liu Zizheng Pan Haoyu He Yuetian Weng Chunhua Shen 31 47 0 02 Feb 2023
Curriculum-Guided Abstractive Summarization Sajad Sotudeh Hanieh Deilamsalehy Franck Dernoncourt Nazli Goharian 35 1 0 02 Feb 2023
Robust Lottery Tickets for Pre-trained Language Models Rui Zheng Rong Bao Yuhao Zhou Di Liang Sirui Wang Wei Wu Tao Gui Qi Zhang Xuanjing Huang AAML 30 13 0 06 Nov 2022
Intriguing Properties of Compression on Multilingual Models Kelechi Ogueji Orevaoghene Ahia Gbemileke Onilude Sebastian Gehrmann Sara Hooker Julia Kreutzer 21 12 0 04 Nov 2022
BEBERT: Efficient and Robust Binary Ensemble BERT Jiayi Tian Chao Fang Hong Wang Zhongfeng Wang MQ 32 16 0 28 Oct 2022
Gradient-based Weight Density Balancing for Robust Dynamic Sparse Training Mathias Parger Alexander Ertl Paul Eibensteiner J. H. Mueller Martin Winter M. Steinberger 34 0 0 25 Oct 2022
Monarch: Expressive Structured Matrices for Efficient and Accurate Training Tri Dao Beidi Chen N. Sohoni Arjun D Desai Michael Poli Jessica Grogan Alexander Liu Aniruddh Rao Atri Rudra Christopher Ré 24 87 0 01 Apr 2022
Structured Pruning Learns Compact and Accurate Models Mengzhou Xia Zexuan Zhong Danqi Chen VLM 9 177 0 01 Apr 2022
When to Prune? A Policy towards Early Structural Pruning Maying Shen Pavlo Molchanov Hongxu Yin J. Álvarez VLM 28 53 0 22 Oct 2021
NormFormer: Improved Transformer Pretraining with Extra Normalization Sam Shleifer Jason Weston Myle Ott AI4CE 33 74 0 18 Oct 2021
Sanity Checks for Lottery Tickets: Does Your Winning Ticket Really Win the Jackpot? Xiaolong Ma Geng Yuan Xuan Shen Tianlong Chen Xuxi Chen ... Ning Liu Minghai Qin Sijia Liu Zhangyang Wang Yanzhi Wang 21 63 0 01 Jul 2021
Playing Lottery Tickets with Vision and Language Zhe Gan Yen-Chun Chen Linjie Li Tianlong Chen Yu Cheng Shuohang Wang Jingjing Liu Lijuan Wang Zicheng Liu VLM 109 54 0 23 Apr 2021
Early-Bird GCNs: Graph-Network Co-Optimization Towards More Efficient GCN Training and Inference via Drawing Early-Bird Lottery Tickets Haoran You Zhihan Lu Zijian Zhou Y. Fu Yingyan Lin GNN 38 30 0 01 Mar 2021
AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning Yuhan Liu Saurabh Agarwal Shivaram Venkataraman OffRL 19 53 0 02 Feb 2021
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 156 345 0 23 Jul 2020
Comparing Rewinding and Fine-tuning in Neural Network Pruning Alex Renda Jonathan Frankle Michael Carbin 229 383 0 05 Mar 2020
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism M. Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 245 1,826 0 17 Sep 2019
Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT Sheng Shen Zhen Dong Jiayu Ye Linjian Ma Z. Yao A. Gholami Michael W. Mahoney Kurt Keutzer MQ 236 576 0 12 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 299 6,984 0 20 Apr 2018