Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

8 February 2023

Papers citing "Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models"

35 / 35 papers shown

Title
Language model compression with weighted low-rank factorization Yen-Chang Hsu Ting Hua Sung-En Chang Qiang Lou Yilin Shen Hongxia Jin 41 102 0 30 Jun 2022
Emergent Abilities of Large Language Models Jason W. Wei Yi Tay Rishi Bommasani Colin Raffel Barret Zoph ... Tatsunori Hashimoto Oriol Vinyals Percy Liang J. Dean W. Fedus ELM ReLM LRM 175 2,428 0 15 Jun 2022
Exploring Extreme Parameter Compression for Pre-trained Language Models Yuxin Ren Benyou Wang Lifeng Shang Xin Jiang Qun Liu 38 18 0 20 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 284 3,583 0 02 May 2022
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model Shaden Smith M. Patwary Brandon Norick P. LeGresley Samyam Rajbhandari ... Mohammad Shoeybi Yuxiong He Michael Houston Saurabh Tiwary Bryan Catanzaro MoE 139 737 0 28 Jan 2022
LaMDA: Language Models for Dialog Applications R. Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha ... Blaise Aguera-Arcas Claire Cui M. Croak Ed H. Chi Quoc Le ALM 103 1,577 0 20 Jan 2022
Scaling Language Models: Methods, Analysis & Insights from Training Gopher Jack W. Rae Sebastian Borgeaud Trevor Cai Katie Millican Jordan Hoffmann ... Jeff Stanway L. Bennett Demis Hassabis Koray Kavukcuoglu G. Irving 68 1,303 0 08 Dec 2021
Direction is what you need: Improving Word Embedding Compression in Large Language Models Klaudia Bałazy Mohammadreza Banaei R. Lebret Jacek Tabor Karl Aberer 45 6 0 15 Jun 2021
Multi-Head Attention: Collaborate Instead of Concatenate Jean-Baptiste Cordonnier Andreas Loukas Martin Jaggi 23 110 0 29 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 507 41,106 0 28 May 2020
LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compression Yihuan Mao Yujing Wang Chufan Wu Chen Zhang Yang-Feng Wang Yaming Yang Quanlu Zhang Yunhai Tong Jing Bai 39 72 0 08 Apr 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 458 4,662 0 23 Jan 2020
PyTorch: An Imperative Style, High-Performance Deep Learning Library Adam Paszke Sam Gross Francisco Massa Adam Lerer James Bradbury ... Sasank Chilamkurthy Benoit Steiner Lu Fang Junjie Bai Soumith Chintala ODL 282 42,038 0 03 Dec 2019
Structured Pruning of Large Language Models Ziheng Wang Jeremy Wohlwend Tao Lei 38 283 0 10 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 136 7,437 0 02 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 272 6,420 0 26 Sep 2019
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 93 588 0 25 Sep 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 62 1,847 0 23 Sep 2019
On the Effectiveness of Low-Rank Matrix Factorization for LSTM Model Compression Genta Indra Winata Andrea Madotto Jamin Shin Elham J. Barezi Pascale Fung 39 28 0 27 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 415 24,160 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 183 8,386 0 19 Jun 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 80 1,051 0 25 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 76 1,120 0 23 May 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 998 93,936 0 11 Oct 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 658 7,080 0 20 Apr 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 484 129,831 0 12 Jun 2017
Pointer Sentinel Mixture Models Stephen Merity Caiming Xiong James Bradbury R. Socher RALM 173 2,814 0 26 Sep 2016
Pruning Filters for Efficient ConvNets Hao Li Asim Kadav Igor Durdanovic H. Samet H. Graf 3DPC 172 3,687 0 31 Aug 2016
Compression of Neural Machine Translation Models via Pruning A. See Minh-Thang Luong Christopher D. Manning MedIm VLM 43 221 0 29 Jun 2016
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients Shuchang Zhou Yuxin Wu Zekun Ni Xinyu Zhou He Wen Yuheng Zou MQ 95 2,080 0 20 Jun 2016
Learning both Weights and Connections for Efficient Neural Networks Song Han Jeff Pool J. Tran W. Dally CVBM 247 6,628 0 08 Jun 2015
Distilling the Knowledge in a Neural Network Geoffrey E. Hinton Oriol Vinyals J. Dean FedML 243 19,523 0 09 Mar 2015
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 882 149,474 0 22 Dec 2014
Compressing Deep Convolutional Networks using Vector Quantization Yunchao Gong Liu Liu Ming Yang Lubomir D. Bourdev MQ 90 1,168 0 18 Dec 2014
Revisiting Natural Gradient for Deep Networks Razvan Pascanu Yoshua Bengio ODL 101 388 0 16 Jan 2013