The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment

13 February 2023

Papers citing "The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment"

25 / 25 papers shown

Title
Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Training Jared Fernandez Luca Wehrstedt Leonid Shamis Mostafa Elhoushi Kalyan Saladi Yonatan Bisk Emma Strubell Jacob Kahn 426 3 0 20 Nov 2024
Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling? Yi Tay Mostafa Dehghani Samira Abnar Hyung Won Chung W. Fedus J. Rao Sharan Narang Vinh Q. Tran Dani Yogatama Donald Metzler AI4CE 79 102 0 21 Jul 2022
The Carbon Footprint of Machine Learning Training Will Plateau, Then Shrink David A. Patterson Joseph E. Gonzalez Urs Holzle Quoc V. Le Chen Liang Lluís-Miquel Munguía D. Rothchild David R. So Maud Texier J. Dean AI4CE 62 239 0 11 Apr 2022
Sustainable AI: Environmental Implications, Challenges and Opportunities Carole-Jean Wu Ramya Raghavendra Udit Gupta Bilge Acun Newsha Ardalani ... Maximilian Balandat Joe Spisak R. Jain Michael G. Rabbat K. Hazelwood 71 396 0 30 Oct 2021
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing Sanyuan Chen Chengyi Wang Zhengyang Chen Yu-Huan Wu Shujie Liu ... Yao Qian Jian Wu Micheal Zeng Xiangzhan Yu Furu Wei SSL 178 1,794 0 26 Oct 2021
The Efficiency Misnomer Daoyuan Chen Liuyi Yao Dawei Gao Ashish Vaswani Yaliang Li 74 101 0 25 Oct 2021
Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers Yi Tay Mostafa Dehghani J. Rao W. Fedus Samira Abnar Hyung Won Chung Sharan Narang Dani Yogatama Ashish Vaswani Donald Metzler 217 112 0 22 Sep 2021
How to Train BERT with an Academic Budget Peter Izsak Moshe Berchansky Omer Levy 77 116 0 15 Apr 2021
Going deeper with Image Transformers Hugo Touvron Matthieu Cord Alexandre Sablayrolles Gabriel Synnaeve Hervé Jégou ViT 121 998 0 31 Mar 2021
DeepViT: Towards Deeper Vision Transformer Daquan Zhou Bingyi Kang Xiaojie Jin Linjie Yang Xiaochen Lian Zihang Jiang Qibin Hou Jiashi Feng ViT 70 517 0 22 Mar 2021
Linformer: Self-Attention with Linear Complexity Sinong Wang Belinda Z. Li Madian Khabsa Han Fang Hao Ma 177 1,678 0 08 Jun 2020
Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training Hongyu Zhu Amar Phanishayee Gennady Pekhimenko 98 50 0 05 Jun 2020
Funnel-Transformer: Filtering out Sequential Redundancy for Efficient Language Processing Zihang Dai Guokun Lai Yiming Yang Quoc V. Le 72 233 0 05 Jun 2020
DynaBERT: Dynamic BERT with Adaptive Width and Depth Lu Hou Zhiqi Huang Lifeng Shang Xin Jiang Xiao Chen Qun Liu MQ 65 322 0 08 Apr 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 93 807 0 06 Apr 2020
HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing Xiyou Zhou Zhiyu Zoey Chen Xiaoyong Jin Wenjie Wang 49 33 0 14 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 462 4,662 0 23 Jan 2020
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 138 7,437 0 02 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 274 6,420 0 26 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 419 24,160 0 26 Jul 2019
Green AI Roy Schwartz Jesse Dodge Noah A. Smith Oren Etzioni 85 1,124 0 22 Jul 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 123 17,950 0 28 May 2019
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware Han Cai Ligeng Zhu Song Han 81 1,865 0 02 Dec 2018
SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size F. Iandola Song Han Matthew W. Moskewicz Khalid Ashraf W. Dally Kurt Keutzer 130 7,448 0 24 Feb 2016
Caffe: Convolutional Architecture for Fast Feature Embedding Yangqing Jia Evan Shelhamer Jeff Donahue Sergey Karayev Jonathan Long Ross B. Girshick S. Guadarrama Trevor Darrell VLM BDL 3DV 213 14,703 0 20 Jun 2014