Centroid-centered Modeling for Efficient Vision Transformer Pre-training

8 March 2023

Bo Du

Papers citing "Centroid-centered Modeling for Efficient Vision Transformer Pre-training"

33 / 33 papers shown

Title
RILS: Masked Visual Reconstruction in Language Semantic Space Shusheng Yang Yixiao Ge Kun Yi Dian Li Ying Shan Xiaohu Qie Xinggang Wang CLIP 67 11 0 17 Jan 2023
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 66 319 0 12 Aug 2022
SdAE: Self-distillated Masked Autoencoder Yabo Chen Yuchen Liu Dongsheng Jiang Xiaopeng Zhang Wenrui Dai H. Xiong Qi Tian ViT 76 73 0 31 Jul 2022
mc-BEiT: Multi-choice Discretization for Image BERT Pre-training Xiaotong Li Yixiao Ge Kun Yi Zixuan Hu Ying Shan Ling-yu Duan 84 39 0 29 Mar 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 97 859 0 07 Feb 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 116 396 0 07 Feb 2022
Are Large-scale Datasets Necessary for Self-Supervised Pre-training? Alaaeldin El-Nouby Gautier Izacard Hugo Touvron Ivan Laptev Hervé Jégou Edouard Grave SSL 89 151 0 20 Dec 2021
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 149 670 0 16 Dec 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 117 244 0 24 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 88 740 0 15 Nov 2021
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 473 7,819 0 11 Nov 2021
VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning Hao Tan Jie Lei Thomas Wolf Joey Tianyi Zhou 91 66 0 21 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 289 2,841 0 15 Jun 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 708 6,127 0 29 Apr 2021
An Empirical Study of Training Self-Supervised Vision Transformers Xinlei Chen Saining Xie Kaiming He ViT 157 1,869 0 05 Apr 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 418 4,996 0 24 Feb 2021
Exploring Simple Siamese Representation Learning Xinlei Chen Kaiming He SSL 258 4,072 0 20 Nov 2020
Unsupervised Learning of Visual Features by Contrasting Cluster Assignments Mathilde Caron Ishan Misra Julien Mairal Priya Goyal Piotr Bojanowski Armand Joulin OCL SSL 253 4,098 0 17 Jun 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 395 6,837 0 13 Jun 2020
Prototypical Contrastive Learning of Unsupervised Representations Junnan Li Pan Zhou Caiming Xiong Guosheng Lin SSL DRL 139 975 0 11 May 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 375 18,866 0 13 Feb 2020
Momentum Contrast for Unsupervised Visual Representation Learning Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross B. Girshick SSL 210 12,124 0 13 Nov 2019
Self-labelling via simultaneous clustering and representation learning Yuki M. Asano Christian Rupprecht Andrea Vedaldi SSL 123 777 0 13 Nov 2019
Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination Zhirong Wu Yuanjun Xiong Stella X. Yu Dahua Lin SSL 182 3,466 0 05 May 2018
Unsupervised Representation Learning by Predicting Image Rotations Spyros Gidaris Praveer Singh N. Komodakis OOD SSL DRL 263 3,298 0 21 Mar 2018
Neural Discrete Representation Learning Aaron van den Oord Oriol Vinyals Koray Kavukcuoglu BDL SSL OCL 230 5,071 0 02 Nov 2017
Colorization as a Proxy Task for Visual Understanding Gustav Larsson Michael Maire Gregory Shakhnarovich SSL 162 498 0 11 Mar 2017
Billion-scale similarity search with GPUs Jeff Johnson Matthijs Douze Hervé Jégou 257 3,737 0 28 Feb 2017
Categorical Reparameterization with Gumbel-Softmax Eric Jang S. Gu Ben Poole BDL 349 5,379 0 03 Nov 2016
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables Chris J. Maddison A. Mnih Yee Whye Teh BDL 198 2,537 0 02 Nov 2016
Discrete Variational Autoencoders J. Rolfe BDL DRL 196 260 0 07 Sep 2016
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 404 1,890 0 18 Aug 2016
Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles M. Noroozi Paolo Favaro SSL 177 8 0 30 Mar 2016