HAT: Hardware-Aware Transformers for Efficient Natural Language Processing

28 May 2020

Zhijian Liu

Chuang Gan

Song Han

ArXiv PDF HTML

Papers citing "HAT: Hardware-Aware Transformers for Efficient Natural Language Processing"

49 / 49 papers shown

Title
Merino: Entropy-driven Design for Generative Language Models on IoT Devices Youpeng Zhao Ming Lin Huadong Tang Qiang Wu Jun Wang 102 0 0 28 Jan 2025
Multi-objective Differentiable Neural Architecture Search R. Sukthanker Arber Zela B. Staffler Samuel Dooley Josif Grabocka Frank Hutter 83 1 0 28 Feb 2024
TransAxx: Efficient Transformers with Approximate Computing Dimitrios Danopoulos Georgios Zervakis Dimitrios Soudris Jörg Henkel ViT 66 2 0 12 Feb 2024
DistDNAS: Search Efficient Feature Interactions within 2 Hours Tunhou Zhang W. Wen Igor Fedorov Xi Liu Buyun Zhang ... Wen-Yen Chen Yiping Han Feng Yan Hai Helen Li Yiran Chen 54 1 0 01 Nov 2023
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers Tobias Christian Nauen Sebastián M. Palacio Federico Raue Andreas Dengel 88 4 0 18 Aug 2023
APQ: Joint Search for Network Architecture, Pruning and Quantization Policy Tianzhe Wang Kuan-Chieh Wang Han Cai Ji Lin Zhijian Liu Song Han MQ 69 174 0 15 Jun 2020
MicroNet for Efficient Language Modeling Zhongxia Yan Hanrui Wang Demi Guo Song Han 42 8 0 16 May 2020
GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learning Hanrui Wang Kuan-Chieh Wang Jiacheng Yang Linxiao Shen Nan Sun Hae-Seung Lee Song Han GNN 54 238 0 30 Apr 2020
Lite Transformer with Long-Short Range Attention Zhanghao Wu Zhijian Liu Ji Lin Chengyue Wu Song Han 53 321 0 24 Apr 2020
A $^3$ : Accelerating Attention Mechanisms in Neural Networks with Approximation Tae Jun Ham Sungjun Jung Seonghak Kim Young H. Oh Yeonhong Park ... Jung-Hun Park Sanghee Lee Kyoung Park Jae W. Lee D. Jeong 50 218 0 22 Feb 2020
SpArch: Efficient Architecture for Sparse Matrix Multiplication Zhekai Zhang Hanrui Wang Song Han W. Dally 48 229 0 20 Feb 2020
Once-for-All: Train One Network and Specialize it for Efficient Deployment Han Cai Chuang Gan Tianzhe Wang Zhekai Zhang Song Han OOD 98 1,277 0 26 Aug 2019
Facebook FAIR's WMT19 News Translation Task Submission Nathan Ng Kyra Yee Alexei Baevski Myle Ott Michael Auli Sergey Edunov VLM 58 396 0 15 Jul 2019
Microsoft Translator at WMT 2019: Towards Large-Scale Document-Level Neural Machine Translation Marcin Junczys-Dowmunt 54 158 0 14 Jul 2019
Syntactically Supervised Transformers for Faster Neural Machine Translation Nader Akoury Kalpesh Krishna Mohit Iyyer 46 55 0 06 Jun 2019
Energy and Policy Considerations for Deep Learning in NLP Emma Strubell Ananya Ganesh Andrew McCallum 62 2,647 0 05 Jun 2019
Imitation Learning for Non-Autoregressive Neural Machine Translation Bingzhen Wei Mingxuan Wang Hao Zhou Junyang Lin Jun Xie Xu Sun 40 94 0 05 Jun 2019
Learning Deep Transformer Models for Machine Translation Qiang Wang Bei Li Tong Xiao Jingbo Zhu Changliang Li Derek F. Wong Lidia S. Chao 70 670 0 05 Jun 2019
Levenshtein Transformer Jiatao Gu Changhan Wang Jake Zhao 107 359 0 27 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 96 1,134 0 23 May 2019
Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin 70 285 0 19 May 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 91 3,147 0 01 Apr 2019
Single Path One-Shot Neural Architecture Search with Uniform Sampling Zichao Guo Xiangyu Zhang Haoyuan Mu Wen Heng Zechun Liu Yichen Wei Jian Sun 64 935 0 31 Mar 2019
The Evolved Transformer David R. So Chen Liang Quoc V. Le ViT 91 462 0 30 Jan 2019
Pay Less Attention with Lightweight and Dynamic Convolutions Felix Wu Angela Fan Alexei Baevski Yann N. Dauphin Michael Auli 70 607 0 29 Jan 2019
FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search Bichen Wu Xiaoliang Dai Peizhao Zhang Yanghan Wang Fei Sun Yiming Wu Yuandong Tian Peter Vajda Yangqing Jia Kurt Keutzer MQ 97 1,300 0 09 Dec 2018
Learning to Design Circuits Hanrui Wang Jiacheng Yang Hae-Seung Lee Song Han 61 93 0 05 Dec 2018
ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware Han Cai Ligeng Zhu Song Han 91 1,867 0 02 Dec 2018
Microsoft's Submission to the WMT2018 News Translation Task: How I Learned to Stop Worrying and Love the Data Marcin Junczys-Dowmunt 70 38 0 01 Sep 2018
Training Deeper Neural Machine Translation Models with Transparent Attention Ankur Bapna Mengzhao Chen Orhan Firat Yuan Cao Yonghui Wu 57 139 0 22 Aug 2018
MnasNet: Platform-Aware Neural Architecture Search for Mobile Mingxing Tan Bo Chen Ruoming Pang Vijay Vasudevan Mark Sandler Andrew G. Howard Quoc V. Le MQ 114 3,004 0 31 Jul 2018
DARTS: Differentiable Architecture Search Hanxiao Liu Karen Simonyan Yiming Yang 185 4,345 0 24 Jun 2018
Marian: Cost-effective High-Quality Neural Machine Translation in C++ Marcin Junczys-Dowmunt Kenneth Heafield Hieu T. Hoang Roman Grundkiewicz Anthony Aue 46 66 0 30 May 2018
Accelerating Neural Transformer via an Average Attention Network Biao Zhang Deyi Xiong Jinsong Su 64 120 0 02 May 2018
The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation Mengzhao Chen Orhan Firat Ankur Bapna Melvin Johnson Wolfgang Macherey ... Niki Parmar M. Schuster Zhifeng Chen Yonghui Wu Macduff Hughes AIMat 52 457 0 26 Apr 2018
A Call for Clarity in Reporting BLEU Scores Matt Post 125 2,978 0 23 Apr 2018
AMC: AutoML for Model Compression and Acceleration on Mobile Devices Yihui He Ji Lin Zhijian Liu Hanrui Wang Li Li Song Han 86 1,348 0 10 Feb 2018
Efficient Neural Architecture Search via Parameter Sharing Hieu H. Pham M. Guan Barret Zoph Quoc V. Le J. Dean 98 2,762 0 09 Feb 2018
Non-Autoregressive Neural Machine Translation Jiatao Gu James Bradbury Caiming Xiong Victor O.K. Li R. Socher 93 795 0 07 Nov 2017
Learning Transferable Architectures for Scalable Image Recognition Barret Zoph Vijay Vasudevan Jonathon Shlens Quoc V. Le 163 5,595 0 21 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 591 130,942 0 12 Jun 2017
Convolutional Sequence to Sequence Learning Jonas Gehring Michael Auli David Grangier Denis Yarats Yann N. Dauphin AIMat 135 3,283 0 08 May 2017
MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications Andrew G. Howard Menglong Zhu Bo Chen Dmitry Kalenichenko Weijun Wang Tobias Weyand M. Andreetto Hartwig Adam 3DH 1.1K 20,781 0 17 Apr 2017
Neural Architecture Search with Reinforcement Learning Barret Zoph Quoc V. Le 416 5,367 0 05 Nov 2016
Efficient softmax approximation for GPUs Edouard Grave Armand Joulin Moustapha Cissé David Grangier Hervé Jégou 73 271 0 14 Sep 2016
SGDR: Stochastic Gradient Descent with Warm Restarts I. Loshchilov Frank Hutter ODL 278 8,091 0 13 Aug 2016
Sequence-Level Knowledge Distillation Yoon Kim Alexander M. Rush 103 1,114 0 25 Jun 2016
Adam: A Method for Stochastic Optimization Diederik P. Kingma Jimmy Ba ODL 1.3K 149,842 0 22 Dec 2014
Convolutional Neural Networks for Sentence Classification Yoon Kim AILaw VLM 586 13,416 0 25 Aug 2014