ConvMAE: Masked Convolution Meets Masked Autoencoders

8 May 2022

Yu Qiao

Papers citing "ConvMAE: Masked Convolution Meets Masked Autoencoders"

50 / 67 papers shown

Title
PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm Haoyi Zhu Honghui Yang Xiaoyang Wu Di Huang Sha Zhang ... Hengshuang Zhao Chunhua Shen Yu Qiao Tong He Wanli Ouyang SSL 103 44 0 12 Oct 2023
Unleashing Vanilla Vision Transformer with Masked Image Modeling for Object Detection Yuxin Fang Shusheng Yang Shijie Wang Yixiao Ge Ying Shan Xinggang Wang 61 56 0 06 Apr 2022
Exploring Plain Vision Transformer Backbones for Object Detection Yanghao Li Hanzi Mao Ross B. Girshick Kaiming He ViT 61 794 0 30 Mar 2022
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Zhan Tong Yibing Song Jue Wang Limin Wang ViT 195 1,156 0 23 Mar 2022
Three things everyone should know about Vision Transformers Hugo Touvron Matthieu Cord Alaaeldin El-Nouby Jakob Verbeek Hervé Jégou ViT 67 121 0 18 Mar 2022
Accelerating DETR Convergence via Semantic-Aligned Matching Gongjie Zhang Zhipeng Luo Yingchen Yu Kaiwen Cui Shijian Lu ViT 75 100 0 14 Mar 2022
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs Xiaohan Ding Xinming Zhang Yi Zhou Jungong Han Guiguang Ding Jian Sun VLM 103 536 0 13 Mar 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 77 845 0 07 Feb 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 73 391 0 07 Feb 2022
DAB-DETR: Dynamic Anchor Boxes are Better Queries for DETR Shilong Liu Feng Li Hao Zhang Xiaohu Yang Xianbiao Qi Hang Su Jun Zhu Lei Zhang ViT 249 740 0 28 Jan 2022
UniFormer: Unifying Convolution and Self-attention for Visual Recognition Kunchang Li Yali Wang Junhao Zhang Peng Gao Guanglu Song Yu Liu Hongsheng Li Yu Qiao ViT 180 372 0 24 Jan 2022
A ConvNet for the 2020s Zhuang Liu Hanzi Mao Chaozheng Wu Christoph Feichtenhofer Trevor Darrell Saining Xie ViT 91 5,102 0 10 Jan 2022
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 130 661 0 16 Dec 2021
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection Yanghao Li Chaoxia Wu Haoqi Fan K. Mangalam Bo Xiong Jitendra Malik Christoph Feichtenhofer ViT 126 685 0 02 Dec 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 83 241 0 24 Nov 2021
Benchmarking Detection Transfer Learning with Vision Transformers Yanghao Li Saining Xie Xinlei Chen Piotr Dollar Kaiming He Ross B. Girshick 53 168 0 22 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 156 1,331 0 18 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 64 722 0 15 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 378 7,600 0 11 Nov 2021
Fast Convergence of DETR with Spatially Modulated Co-Attention Peng Gao Minghang Zheng Xiaogang Wang Jifeng Dai Hongsheng Li ViT 64 305 0 05 Aug 2021
Early Convolutions Help Transformers See Better Tete Xiao Mannat Singh Eric Mintun Trevor Darrell Piotr Dollár Ross B. Girshick 37 763 0 28 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 182 2,790 0 15 Jun 2021
CoAtNet: Marrying Convolution and Attention for All Data Sizes Zihang Dai Hanxiao Liu Quoc V. Le Mingxing Tan ViT 93 1,188 0 09 Jun 2021
Container: Context Aggregation Network Peng Gao Jiasen Lu Hongsheng Li Roozbeh Mottaghi Aniruddha Kembhavi ViT 52 71 0 02 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 587 5,920 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 132 1,837 0 05 Apr 2021
CvT: Introducing Convolutions to Vision Transformers Haiping Wu Bin Xiao Noel Codella Mengchen Liu Xiyang Dai Lu Yuan Lei Zhang ViT 116 1,891 0 29 Mar 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 351 21,175 0 25 Mar 2021
ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biases Stéphane dÁscoli Hugo Touvron Matthew L. Leavitt Ari S. Morcos Giulio Biroli Levent Sagun ViT 97 818 0 19 Mar 2021
Transformer in Transformer Kai Han An Xiao Enhua Wu Jianyuan Guo Chunjing Xu Yunhe Wang ViT 362 1,544 0 27 Feb 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 703 28,659 0 26 Feb 2021
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions Wenhai Wang Enze Xie Xiang Li Deng-Ping Fan Kaitao Song Ding Liang Tong Lu Ping Luo Ling Shao ViT 459 3,678 0 24 Feb 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 329 4,873 0 24 Feb 2021
Conditional Positional Encodings for Vision Transformers Xiangxiang Chu Zhi Tian Bo Zhang Xinlong Wang Chunhua Shen ViT 62 610 0 22 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 332 2,016 0 09 Feb 2021
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet Li-xin Yuan Yunpeng Chen Tao Wang Weihao Yu Yujun Shi Zihang Jiang Francis E. H. Tay Jiashi Feng Shuicheng Yan ViT 104 1,918 0 28 Jan 2021
Bottleneck Transformers for Visual Recognition A. Srinivas Nayeon Lee Niki Parmar Jonathon Shlens Pieter Abbeel Ashish Vaswani SLR 325 986 0 27 Jan 2021
Training data-efficient image transformers & distillation through attention Hugo Touvron Matthieu Cord Matthijs Douze Francisco Massa Alexandre Sablayrolles Hervé Jégou ViT 317 6,657 0 23 Dec 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 425 40,217 0 22 Oct 2020
Deformable DETR: Deformable Transformers for End-to-End Object Detection Xizhou Zhu Weijie Su Lewei Lu Bin Li Xiaogang Wang Jifeng Dai ViT 174 4,993 0 08 Oct 2020
Locally Masked Convolution for Autoregressive Models Ajay Jain Pieter Abbeel Deepak Pathak DiffM OffRL 58 31 0 22 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 526 41,106 0 28 May 2020
End-to-End Object Detection with Transformers Nicolas Carion Francisco Massa Gabriel Synnaeve Nicolas Usunier Alexander Kirillov Sergey Zagoruyko ViT 3DV PINN 312 12,906 0 26 May 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 275 18,607 0 13 Feb 2020
Momentum Contrast for Unsupervised Visual Representation Learning Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross B. Girshick SSL 124 12,007 0 13 Nov 2019
LXMERT: Learning Cross-Modality Encoder Representations from Transformers Hao Hao Tan Joey Tianyi Zhou VLM MLLM 209 2,467 0 20 Aug 2019
Dynamic Fusion with Intra- and Inter- Modality Attention Flow for Visual Question Answering Peng Gao Zhengkai Jiang Haoxuan You Pan Lu Steven C. H. Hoi Xiaogang Wang Hongsheng Li AIMat 69 364 0 13 Dec 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.1K 93,936 0 11 Oct 2018
Unified Perceptual Parsing for Scene Understanding Tete Xiao Yingcheng Liu Bolei Zhou Yuning Jiang Jian Sun OCL VOS 133 1,859 0 26 Jul 2018
Image Inpainting for Irregular Holes Using Partial Convolutions Guilin Liu F. Reda Kevin J. Shih Ting-Chun Wang Andrew Tao Bryan Catanzaro 199 1,919 0 20 Apr 2018