Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks

8 June 2022

Papers citing "Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Tasks"

23 / 23 papers shown

Title
Towards Understanding Why FixMatch Generalizes Better Than Supervised Learning Jingyang Li Jiachun Pan Vincent Y. F. Tan Kim-Chuan Toh Pan Zhou AAML MLT 112 0 0 15 Oct 2024
Masked Image Modeling: A Survey Vlad Hondru Florinel-Alin Croitoru Shervin Minaee Radu Tudor Ionescu N. Sebe 104 8 0 13 Aug 2024
Masked Siamese ConvNets L. Jing Jiachen Zhu Yann LeCun SSL 79 35 0 15 Jun 2022
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language Alexei Baevski Wei-Ning Hsu Qiantong Xu Arun Babu Jiatao Gu Michael Auli SSL VLM ViT 77 845 0 07 Feb 2022
Masked Feature Prediction for Self-Supervised Visual Pre-Training Chen Wei Haoqi Fan Saining Xie Chaoxia Wu Alan Yuille Christoph Feichtenhofer ViT 130 661 0 16 Dec 2021
PeCo: Perceptual Codebook for BERT Pre-training of Vision Transformers Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu Baining Guo ViT 83 241 0 24 Nov 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 179 2,790 0 15 Jun 2021
Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss Jeff Z. HaoChen Colin Wei Adrien Gaidon Tengyu Ma SSL 46 306 0 08 Jun 2021
Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learning Zixin Wen Yuanzhi Li SSL MLT 47 133 0 31 May 2021
Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning Zeyuan Allen-Zhu Yuanzhi Li FedML 106 362 0 17 Dec 2020
For self-supervised learning, Rationality implies generalization, provably Yamini Bansal Gal Kaplun Boaz Barak OOD SSL 81 22 0 16 Oct 2020
Bootstrap your own latent: A new approach to self-supervised Learning Jean-Bastien Grill Florian Strub Florent Altché Corentin Tallec Pierre Harvey Richemond ... M. G. Azar Bilal Piot Koray Kavukcuoglu Rémi Munos Michal Valko SSL 282 6,718 0 13 Jun 2020
What Makes for Good Views for Contrastive Learning? Yonglong Tian Chen Sun Ben Poole Dilip Krishnan Cordelia Schmid Phillip Isola SSL 78 1,313 0 20 May 2020
Feature Purification: How Adversarial Training Performs Robust Deep Learning Zeyuan Allen-Zhu Yuanzhi Li MLT AAML 61 149 0 20 May 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 429 3,397 0 09 Mar 2020
ClusterFit: Improving Generalization of Visual Representations Xueting Yan Ishan Misra Abhinav Gupta Deepti Ghadiyaram D. Mahajan SSL VLM 84 132 0 06 Dec 2019
EfficientDet: Scalable and Efficient Object Detection Mingxing Tan Ruoming Pang Quoc V. Le 88 4,996 0 20 Nov 2019
Momentum Contrast for Unsupervised Visual Representation Learning Kaiming He Haoqi Fan Yuxin Wu Saining Xie Ross B. Girshick SSL 119 12,007 0 13 Nov 2019
Learning Representations by Maximizing Mutual Information Across Views Philip Bachman R. Devon Hjelm William Buchwalter SSL 167 1,463 0 03 Jun 2019
Unsupervised Deep Learning by Neighbourhood Discovery Jiabo Huang Qi Dong S. Gong Xiatian Zhu SSL 62 152 0 25 Apr 2019
Aggregated Residual Transformations for Deep Neural Networks Saining Xie Ross B. Girshick Piotr Dollár Zhuowen Tu Kaiming He 442 10,281 0 16 Nov 2016
Learning Deep Features for Discriminative Localization Bolei Zhou A. Khosla Àgata Lapedriza A. Oliva Antonio Torralba SSL SSeg FAtt 181 9,280 0 14 Dec 2015
SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation Vijay Badrinarayanan Alex Kendall R. Cipolla SSeg 814 15,718 0 02 Nov 2015