SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs

28 January 2025

Mohammad Mozaffari

Amir Yazdanbakhsh

Papers citing "SLoPe: Double-Pruned Sparse Plus Lazy Low-Rank Adapter Pretraining of LLMs"

12 / 12 papers shown

Title
Accelerating Transformer Inference and Training with 2:4 Activation Sparsity Daniel Haziza Timothy Chou Dhruv Choudhary Luca Wehrstedt Francisco Massa Jiecao Yu Geonhwa Jeong Supriya Rao Patrick Labatut Jesse Cai 44 0 0 20 Mar 2025
CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation Zhengwu Liu Ruijie Zhang Zhilin Wang Zi Yang Paul Hovland Bogdan Nicolae Franck Cappello Z. Zhang 51 0 0 16 Feb 2025
A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models Cong Guo Feng Cheng Zhixu Du James Kiessling Jonathan Ku ... Qilin Zheng Guanglei Zhou Hai Li-Wei Li Yiran Chen 33 7 0 08 Oct 2024
OATS: Outlier-Aware Pruning Through Sparse and Low Rank Decomposition Stephen Zhang Vardan Papyan VLM 51 1 0 20 Sep 2024
ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterization Haoran You Yipin Guo Yichao Fu Wei Zhou Huihong Shi Xiaofan Zhang Souvik Kundu Amir Yazdanbakhsh Y. Lin KELM 59 7 0 10 Jun 2024
Accelerating Transformer Pre-training with 2:4 Sparsity Yuezhou Hu Kang Zhao Weiyu Huang Jianfei Chen Jun Zhu 65 7 0 02 Apr 2024
Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers Abhimanyu Bambhaniya Amir Yazdanbakhsh Suvinay Subramanian Sheng-Chun Kao Shivani Agrawal Utku Evci Tushar Krishna 54 14 0 07 Feb 2024
Training Recipe for N:M Structured Sparsity with Decaying Pruning Mask Sheng-Chun Kao Amir Yazdanbakhsh Suvinay Subramanian Shivani Agrawal Utku Evci T. Krishna 50 12 0 15 Sep 2022
Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks Itay Hubara Brian Chmiel Moshe Island Ron Banner S. Naor Daniel Soudry 59 111 0 16 Feb 2021
Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks Torsten Hoefler Dan Alistarh Tal Ben-Nun Nikoli Dryden Alexandra Peste MQ 153 685 0 31 Jan 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 282 2,007 0 31 Dec 2020
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 304 6,996 0 20 Apr 2018