Accelerating Transformer Pre-training with 2:4 Sparsity

Accelerating Transformer Pre-training with 2:4 Sparsity

2 April 2024

Yuezhou Hu

Kang Zhao

Jianfei Chen

Jun Zhu

Papers citing "Accelerating Transformer Pre-training with 2:4 Sparsity"

6 / 6 papers shown

Title
SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs Mohammad Mozaffari Amir Yazdanbakhsh Zhao Zhang M. Dehnavi 78 5 0 28 Jan 2025
S-STE: Continuous Pruning Function for Efficient 2:4 Sparse Pre-training Yuezhou Hu Jun-Jie Zhu Jianfei Chen 38 0 0 13 Sep 2024
SparseDM: Toward Sparse Efficient Diffusion Models Kafeng Wang Jianfei Chen He Li Zhenpeng Mi Jun-Jie Zhu DiffM 65 8 0 16 Apr 2024
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 50 110 0 16 Feb 2021
The Lottery Ticket Hypothesis for Pre-trained BERT Networks Tianlong Chen Jonathan Frankle Shiyu Chang Sijia Liu Yang Zhang Zhangyang Wang Michael Carbin 153 345 0 23 Jul 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018