Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models

16 January 2025

Papers citing "Optimization Strategies for Enhancing Resource Efficiency in Transformers & Large Language Models"

14 / 14 papers shown

Title
Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning Mengzhou Xia Tianyu Gao Zhiyuan Zeng Danqi Chen 115 300 0 10 Oct 2023
Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiences Fred Hohman Mary Beth Kery Donghao Ren Dominik Moritz 60 17 0 06 Oct 2023
SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression Tim Dettmers Ruslan Svirschevski Vage Egiazarian Denis Kuznedelev Elias Frantar Saleh Ashkboos Alexander Borzunov Torsten Hoefler Dan Alistarh MQ 62 252 0 05 Jun 2023
QLoRA: Efficient Finetuning of Quantized LLMs Tim Dettmers Artidoro Pagnoni Ari Holtzman Luke Zettlemoyer ALM 147 2,555 0 23 May 2023
SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot Elias Frantar Dan Alistarh VLM 93 710 0 02 Jan 2023
The case for 4-bit precision: k-bit Inference Scaling Laws Tim Dettmers Luke Zettlemoyer MQ 88 228 0 19 Dec 2022
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Tim Dettmers M. Lewis Younes Belkada Luke Zettlemoyer MQ 90 653 0 15 Aug 2022
8-bit Optimizers via Block-wise Quantization Tim Dettmers M. Lewis Sam Shleifer Luke Zettlemoyer MQ 112 297 0 06 Oct 2021
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 140 1,897 0 08 Sep 2021
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 176 4,418 0 07 Sep 2020
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 232 7,504 0 02 Oct 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 100 1,061 0 25 May 2019
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge Peter Clark Isaac Cowhey Oren Etzioni Tushar Khot Ashish Sabharwal Carissa Schoenick Oyvind Tafjord ELM RALM LRM 158 2,587 0 14 Mar 2018
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 311 2,859 0 26 Sep 2016