MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity

29 July 2024

Papers citing "MimiQ: Low-Bit Data-Free Quantization of Vision Transformers with Encouraging Inter-Head Attention Similarity"

46 / 46 papers shown

Title
Semantics Prompting Data-Free Quantization for Low-Bit Vision Transformers Mingliang Xu Yuyao Zhou Yuxin Zhang Shen Li Yong Li Rongrong Ji Zhanpeng Zeng Rongrong Ji MQ 128 0 0 31 Dec 2024
CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs Akshat Ramachandran Souvik Kundu Tushar Krishna MQ 71 11 0 07 Jul 2024
Adaptive Data-Free Quantization Biao Qian Yang Wang Richang Hong Meng Wang MQ 39 38 0 13 Mar 2023
RepQ-ViT: Scale Reparameterization for Post-Training Quantization of Vision Transformers Zhikai Li Junrui Xiao Lianwei Yang Qingyi Gu MQ 36 85 0 16 Dec 2022
Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer Yanjing Li Sheng Xu Baochang Zhang Xianbin Cao Penglei Gao Guodong Guo MQ ViT 47 92 0 13 Oct 2022
PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers Zhikai Li Mengjuan Chen Junrui Xiao Qingyi Gu ViT MQ 70 34 0 13 Sep 2022
I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference Zhikai Li Qingyi Gu MQ 87 101 0 04 Jul 2022
It's All In the Teacher: Zero-Shot Quantization Brought Closer to the Teacher Kanghyun Choi Hye Yoon Lee Deokki Hong Joonsang Yu Noseong Park Youngsok Kim Jinho Lee MQ 43 33 0 31 Mar 2022
Patch Similarity Aware Data-Free Quantization for Vision Transformers Zhikai Li Liping Ma Mengjuan Chen Junrui Xiao Qingyi Gu MQ ViT 48 45 0 04 Mar 2022
SPViT: Enabling Faster Vision Transformers via Soft Token Pruning Zhenglun Kong Peiyan Dong Xiaolong Ma Xin Meng Mengshu Sun ... Geng Yuan Bin Ren Minghai Qin Hao Tang Yanzhi Wang ViT 39 145 0 27 Dec 2021
A Unified Pruning Framework for Vision Transformers Hao Yu Jianxin Wu ViT 30 62 0 30 Nov 2021
FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer Yang Lin Tianyu Zhang Peiqin Sun Zheng Li Shuchang Zhou ViT MQ 33 150 0 27 Nov 2021
PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantization Zhihang Yuan Chenhao Xue Yiqi Chen Qiang Wu Guangyu Sun ViT MQ 33 136 0 24 Nov 2021
IntraQ: Learning Synthetic Images with Intra-Class Heterogeneity for Zero-Shot Network Quantization Mingliang Xu Mingbao Lin Gongrui Nan Jianzhuang Liu Baochang Zhang Yonghong Tian Rongrong Ji MQ 60 73 0 17 Nov 2021
Qimera: Data-free Quantization with Synthetic Boundary Supporting Samples Kanghyun Choi Deokki Hong Noseong Park Youngsok Kim Jinho Lee MQ 38 65 0 04 Nov 2021
Post-Training Quantization for Vision Transformer Zhenhua Liu Yunhe Wang Kai Han Siwei Ma Wen Gao ViT MQ 76 332 0 27 Jun 2021
A White Paper on Neural Network Quantization Markus Nagel Marios Fournarakis Rana Ali Amjad Yelysei Bondarenko M. V. Baalen Tijmen Blankevoort MQ 46 518 0 15 Jun 2021
AutoReCon: Neural Architecture Search-based Reconstruction for Data-free Compression Baozhou Zhu P. Hofstee J. Peltenburg Jinho Lee Zaid Al-Ars 29 23 0 25 May 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 167 21,051 0 25 Mar 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 181 6,657 0 23 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 79 40,217 0 22 Oct 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 261 41,106 0 28 May 2020
Up or Down? Adaptive Rounding for Post-Training Quantization Markus Nagel Rana Ali Amjad M. V. Baalen Christos Louizos Tijmen Blankevoort MQ 25 563 0 22 Apr 2020
Generative Low-bitwidth Data Free Quantization Shoukai Xu Haokun Li Bohan Zhuang Jing Liu Jingyun Liang Chuangrun Liang Mingkui Tan MQ 18 127 0 07 Mar 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 93 18,523 0 13 Feb 2020
ZeroQ: A Novel Zero Shot Quantization Framework Yaohui Cai Z. Yao Zhen Dong A. Gholami Michael W. Mahoney Kurt Keutzer MQ 50 393 0 01 Jan 2020
Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion Hongxu Yin Pavlo Molchanov Zhizhong Li J. Álvarez Arun Mallya Derek Hoiem N. Jha Jan Kautz 37 558 0 18 Dec 2019
MMDetection: Open MMLab Detection Toolbox and Benchmark Kai-xiang Chen Jiaqi Wang Jiangmiao Pang Yuhang Cao Yu Xiong ... Jingdong Wang Jianping Shi Wanli Ouyang Chen Change Loy Dahua Lin VOS 72 2,845 0 17 Jun 2019
Data-Free Quantization Through Weight Equalization and Bias Correction Markus Nagel M. V. Baalen Tijmen Blankevoort Max Welling MQ 24 505 0 11 Jun 2019
Learned Step Size Quantization S. K. Esser J. McKinstry Deepika Bablani R. Appuswamy D. Modha MQ 42 792 0 21 Feb 2019
Low-bit Quantization of Neural Networks for Efficient Inference Yoni Choukroun Eli Kravchik Fan Yang P. Kisilev MQ 40 358 0 18 Feb 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 416 93,936 0 11 Oct 2018
Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference Benoit Jacob S. Kligys Bo Chen Menglong Zhu Matthew Tang Andrew G. Howard Hartwig Adam Dmitry Kalenichenko MQ 108 3,090 0 15 Dec 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 180 129,831 0 12 Jun 2017
Mask R-CNN Kaiming He Georgia Gkioxari Piotr Dollár Ross B. Girshick ObjD 215 27,018 0 20 Mar 2017
Deep Learning with Low Precision by Half-wave Gaussian Quantization Zhaowei Cai Xiaodong He Jian Sun Nuno Vasconcelos MQ 73 504 0 03 Feb 2017
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization Ramprasaath R. Selvaraju Michael Cogswell Abhishek Das Ramakrishna Vedantam Devi Parikh Dhruv Batra FAtt 134 19,796 0 07 Oct 2016
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations Itay Hubara Matthieu Courbariaux Daniel Soudry Ran El-Yaniv Yoshua Bengio MQ 70 1,852 0 22 Sep 2016
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 280 1,850 0 18 Aug 2016
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients Shuchang Zhou Yuxin Wu Zekun Ni Xinyu Zhou He Wen Yuheng Zou MQ 85 2,080 0 20 Jun 2016
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks Mohammad Rastegari Vicente Ordonez Joseph Redmon Ali Farhadi MQ 114 4,342 0 16 Mar 2016
Resiliency of Deep Neural Networks under Quantization Wonyong Sung Sungho Shin Kyuyeon Hwang MQ 35 157 0 20 Nov 2015
Fixed Point Quantization of Deep Convolutional Networks D. Lin S. Talathi V. Annapureddy MQ 51 814 0 19 Nov 2015
BinaryConnect: Training Deep Neural Networks with binary weights during propagations Matthieu Courbariaux Yoshua Bengio J. David MQ 67 2,976 0 02 Nov 2015
Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding Song Han Huizi Mao W. Dally 3DGS 126 8,793 0 01 Oct 2015
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 98 43,290 0 01 May 2014