v1v2 (latest)

Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training

12 June 2023

Lorenzo Baraldi

Lorenzo Baraldi

Papers citing "Learning to Mask and Permute Visual Tokens for Vision Transformer Pre-Training"

37 / 37 papers shown

Title
A Survey of the Self Supervised Learning Mechanisms for Vision Transformers Asifullah Khan A. Sohail Mustansar Fiaz Mehdi Hassan Tariq Habib Afridi ... Muhammad Zaigham Zaheer Kamran Ali Tangina Sultana Ziaurrehman Tanoli Naeem Akhter 229 5 0 30 Aug 2024
Masked Modeling for Self-supervised Representation Learning on Vision and Beyond Siyuan Li Luyuan Zhang Zedong Wang Di Wu Lirong Wu ... Jun Xia Cheng Tan Yang Liu Baigui Sun Stan Z. Li SSL 101 15 0 31 Dec 2023
CAE v2: Context Autoencoder with CLIP Target Xinyu Zhang Jiahui Chen Junkun Yuan Qiang Chen Jian Wang ... Jimin Pi Kun Yao Junyu Han Errui Ding Jingdong Wang VLM CLIP 91 24 0 17 Nov 2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 66 319 0 12 Aug 2022
MILAN: Masked Image Pretraining on Language Assisted Representation Zejiang Hou Fei Sun Yen-kuang Chen Yuan Xie S. Kung ViT 75 68 0 11 Aug 2022
Contrastive Masked Autoencoders are Stronger Vision Learners Zhicheng Huang Xiaojie Jin Cheng Lu Qibin Hou Mingg-Ming Cheng Dongmei Fu Xiaohui Shen Jiashi Feng 119 152 0 27 Jul 2022
MVP: Multimodality-guided Visual Pre-training Longhui Wei Lingxi Xie Wen-gang Zhou Houqiang Li Qi Tian 54 107 0 10 Mar 2022
Corrupted Image Modeling for Self-Supervised Visual Pre-Training Yuxin Fang Li Dong Hangbo Bao Xinggang Wang Furu Wei 66 88 0 07 Feb 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 105 396 0 07 Feb 2022
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? Alaaeldin El-Nouby Gautier Izacard Hugo Touvron Ivan Laptev Hervé Jégou Edouard Grave SSL 86 151 0 20 Dec 2021
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 149 670 0 16 Dec 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 115 244 0 24 Nov 2021
SimMIM: A Simple Framework for Masked Image Modeling Zhenda Xie Zheng Zhang Yue Cao Yutong Lin Jianmin Bao Zhuliang Yao Qi Dai Han Hu 197 1,360 0 18 Nov 2021
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 217 1,822 0 18 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 467 7,814 0 11 Nov 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 281 2,826 0 15 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 694 6,121 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 157 1,868 0 05 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 967 29,731 0 26 Feb 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 418 4,987 0 24 Feb 2021
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 258 4,067 0 20 Nov 2020
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 667 41,369 0 22 Oct 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 371 6,833 0 13 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 826 42,332 0 28 May 2020
MPNet: Masked and Permuted Pre-training for Language Understanding Kaitao Song Xu Tan Tao Qin Jianfeng Lu Tie-Yan Liu 106 1,121 0 20 Apr 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 375 18,778 0 13 Feb 2020
Momentum Contrast for Unsupervised Visual Representation Learning Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross B. Girshick SSL 207 12,085 0 13 Nov 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 668 24,528 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 232 8,444 0 19 Jun 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,114 0 11 Oct 2018
Representation Learning with Contrastive Predictive Coding Aaron van den Oord Yazhe Li Oriol Vinyals DRL SSL 348 10,349 0 10 Jul 2018
Unsupervised Representation Learning by Predicting Image Rotations Spyros Gidaris Praveer Singh N. Komodakis OOD SSL DRL 261 3,297 0 21 Mar 2018
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 404 1,889 0 18 Aug 2016
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles M. Noroozi Paolo Favaro SSL 174 5 0 30 Mar 2016
Colorful Image Colorization Richard Y. Zhang Phillip Isola Alexei A. Efros 149 3,532 0 28 Mar 2016
Unsupervised Visual Representation Learning by Context Prediction Carl Doersch Abhinav Gupta Alexei A. Efros DRL SSL 169 2,789 0 19 May 2015
Fine-Grained Visual Classification of Aircraft Subhransu Maji Esa Rahtu Arno Solin Matthew Blaschko Andrea Vedaldi 123 2,269 0 21 Jun 2013