Training with Quantization Noise for Extreme Model Compression

15 April 2020

Angela Fan

Benjamin Graham

Remi Gribonval

Papers citing "Training with Quantization Noise for Extreme Model Compression"

50 / 67 papers shown

Title
HadamRNN: Binary and Sparse Ternary Orthogonal RNNs Armand Foucault Franck Mamalet François Malgouyres MQ 129 0 0 28 Jan 2025
On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance Jaskirat Singh Bram Adams Ahmed E. Hassan VLM 56 0 0 01 Nov 2024
Data Generation for Hardware-Friendly Post-Training Quantization Lior Dikstein Ariel Lapid Arnon Netzer H. Habi MQ 337 0 0 29 Oct 2024
On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performance Jaskirat Singh Emad Fallahzadeh Bram Adams Ahmed E. Hassan MQ 53 3 0 25 Mar 2024
GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Design Sung Une Lee Boming Xia Yongan Zhang Ang Li Yingyan Lin GNN 74 48 0 22 Dec 2021
AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search Daoyuan Chen Yaliang Li Minghui Qiu Zhen Wang Bofang Li Bolin Ding Hongbo Deng Jun Huang Wei Lin Jingren Zhou MQ 36 104 0 13 Jan 2020
Compressive Transformers for Long-Range Sequence Modelling Jack W. Rae Anna Potapenko Siddhant M. Jayakumar Timothy Lillicrap RALM VLM KELM 34 636 0 13 Nov 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 78 7,386 0 02 Oct 2019
Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks Yuhang Li Xin Dong Wei Wang MQ 40 255 0 28 Sep 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 230 6,420 0 26 Sep 2019
Extremely Small BERT Models from Mixed-Vocabulary Training Sanqiang Zhao Raghav Gupta Yang Song Denny Zhou VLM 33 53 0 25 Sep 2019
Reducing Transformer Depth on Demand with Structured Dropout Angela Fan Edouard Grave Armand Joulin 83 586 0 25 Sep 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 32 1,838 0 23 Sep 2019
Patient Knowledge Distillation for BERT Model Compression S. Sun Yu Cheng Zhe Gan Jingjing Liu 92 833 0 25 Aug 2019
Well-Read Students Learn Better: On the Importance of Pre-training Compact Models Iulia Turc Ming-Wei Chang Kenton Lee Kristina Toutanova 40 224 0 23 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 336 24,160 0 26 Jul 2019
And the Bit Goes Down: Revisiting the Quantization of Neural Networks Pierre Stock Armand Joulin Rémi Gribonval Benjamin Graham Hervé Jégou MQ 49 149 0 12 Jul 2019
Augmenting Self-attention with Persistent Memory Sainbayar Sukhbaatar Edouard Grave Guillaume Lample Hervé Jégou Armand Joulin RALM KELM 43 136 0 02 Jul 2019
A Tensorized Transformer for Language Modeling Xindian Ma Peng Zhang Shuai Zhang Nan Duan Yuexian Hou D. Song M. Zhou 27 165 0 24 Jun 2019
EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks Mingxing Tan Quoc V. Le 3DV MedIm 38 17,950 0 28 May 2019
Adaptive Attention Span in Transformers Sainbayar Sukhbaatar Edouard Grave Piotr Bojanowski Armand Joulin 44 285 0 19 May 2019
Searching for MobileNetV3 Andrew G. Howard Mark Sandler Grace Chu Liang-Chieh Chen Bo Chen ... Yukun Zhu Ruoming Pang Vijay Vasudevan Quoc V. Le Hartwig Adam 242 6,685 0 06 May 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 61 3,141 0 01 Apr 2019
Pay Less Attention with Lightweight and Dynamic Convolutions Felix Wu Angela Fan Alexei Baevski Yann N. Dauphin Michael Auli 34 606 0 29 Jan 2019
Cross-lingual Language Model Pretraining Guillaume Lample Alexis Conneau 41 2,727 0 22 Jan 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 101 3,707 0 09 Jan 2019
Rethinking the Value of Network Pruning Zhuang Liu Mingjie Sun Tinghui Zhou Gao Huang Trevor Darrell 21 1,460 0 11 Oct 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 751 93,936 0 11 Oct 2018
Adaptive Input Representations for Neural Language Modeling Alexei Baevski Michael Auli 77 389 0 28 Sep 2018
ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design Ningning Ma Xiangyu Zhang Haitao Zheng Jian Sun 108 4,957 0 30 Jul 2018
Universal Transformers Mostafa Dehghani Stephan Gouws Oriol Vinyals Jakob Uszkoreit Lukasz Kaiser 58 752 0 10 Jul 2018
Quantizing deep convolutional networks for efficient inference: A whitepaper Raghuraman Krishnamoorthi MQ 96 1,009 0 21 Jun 2018
PACT: Parameterized Clipping Activation for Quantized Neural Networks Jungwook Choi Zhuo Wang Swagath Venkataramani P. Chuang Vijayalakshmi Srinivasan K. Gopalakrishnan MQ 32 945 0 16 May 2018
Accelerating Neural Transformer via an Average Attention Network Biao Zhang Deyi Xiong Jinsong Su 43 120 0 02 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 511 7,080 0 20 Apr 2018
Training wide residual networks for deployment using a single bit for each weight Mark D Mcdonnell MQ 52 71 0 23 Feb 2018
Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks Deepak Mittal S. Bhardwaj Mitesh M. Khapra Balaraman Ravindran VLM 47 65 0 31 Jan 2018
MobileNetV2: Inverted Residuals and Linear Bottlenecks Mark Sandler Andrew G. Howard Menglong Zhu A. Zhmoginov Liang-Chieh Chen 101 19,124 0 13 Jan 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 110 3,090 0 15 Dec 2017
Learning Sparse Neural Networks through $L_0$ Regularization Christos Louizos Max Welling Diederik P. Kingma 151 1,136 0 04 Dec 2017
CondenseNet: An Efficient DenseNet using Learned Group Convolutions Gao Huang Shichen Liu Laurens van der Maaten Kilian Q. Weinberger 61 796 0 25 Nov 2017
ThiNet: A Filter Level Pruning Method for Deep Neural Network Compression Jian-Hao Luo Jianxin Wu Weiyao Lin 30 1,751 0 20 Jul 2017
Model compression as constrained optimization, with application to neural nets. Part II: quantization M. A. Carreira-Perpiñán Yerlan Idelbayev MQ 39 37 0 13 Jul 2017
ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices Xiangyu Zhang Xinyu Zhou Mengxiao Lin Jian Sun AI4TS 106 6,830 0 04 Jul 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 278 129,831 0 12 Jun 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 330 4,444 0 18 Apr 2017
Variational Dropout Sparsifies Deep Neural Networks Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov BDL 61 825 0 19 Jan 2017
Language Modeling with Gated Convolutional Networks Yann N. Dauphin Angela Fan Michael Auli David Grangier 155 2,377 0 23 Dec 2016
FastText.zip: Compressing text classification models Armand Joulin Edouard Grave Piotr Bojanowski Matthijs Douze Hervé Jégou Tomas Mikolov MQ 55 1,201 0 12 Dec 2016
Quasi-Recurrent Neural Networks James Bradbury Stephen Merity Caiming Xiong R. Socher 66 437 0 05 Nov 2016