An Empirical Study of Training Self-Supervised Vision Transformers

5 April 2021

Papers citing "An Empirical Study of Training Self-Supervised Vision Transformers"

50 / 470 papers shown

Title
Deep Spectral Methods: A Surprisingly Strong Baseline for Unsupervised Semantic Segmentation and Localization Luke Melas-Kyriazi Christian Rupprecht Iro Laina Andrea Vedaldi 30 160 0 16 May 2022
The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning Zixin Wen Yuanzhi Li SSL 27 34 0 12 May 2022
Scene Consistency Representation Learning for Video Scene Segmentation Haoqian Wu Keyu Chen Yanan Luo Ruizhi Qiao Bo Ren Haozhe Liu Weicheng Xie Linlin Shen SSL 45 16 0 11 May 2022
Learning to Answer Visual Questions from Web Videos Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid ViT 37 33 0 10 May 2022
ConvMAE: Masked Convolution Meets Masked Autoencoders Peng Gao Teli Ma Hongsheng Li Ziyi Lin Jifeng Dai Yu Qiao ViT 19 121 0 08 May 2022
Relational Representation Learning in Visually-Rich Documents Xin Li Yan Zheng Yiqing Hu H. Cao Yunfei Wu Deqiang Jiang Yinsong Liu Bo Ren 20 12 0 05 May 2022
Better plain ViT baselines for ImageNet-1k Lucas Beyer Xiaohua Zhai Alexander Kolesnikov ViT VLM 33 111 0 03 May 2022
RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning Xiaojian Ma Weili Nie Zhiding Yu Huaizu Jiang Chaowei Xiao Yuke Zhu Song-Chun Zhu Anima Anandkumar ViT LRM 30 19 0 24 Apr 2022
Neuro-BERT: Rethinking Masked Autoencoding for Self-supervised Neurological Pretraining Di Wu Siyuan Li Jie Yang Mohamad Sawan SSL 36 14 0 20 Apr 2022
ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models Chunyuan Li Haotian Liu Liunian Harold Li Pengchuan Zhang J. Aneja ... Ping Jin Houdong Hu Zicheng Liu Yong Jae Lee Jianfeng Gao 38 145 0 19 Apr 2022
Masked Siamese Networks for Label-Efficient Learning Mahmoud Assran Mathilde Caron Ishan Misra Piotr Bojanowski Florian Bordes Pascal Vincent Armand Joulin Michael G. Rabbat Nicolas Ballas SSL 31 310 0 14 Apr 2022
Evaluating Vision Transformer Methods for Deep Reinforcement Learning from Pixels Tianxin Tao Daniele Reda M. van de Panne ViT 11 19 0 11 Apr 2022
Representation Learning by Detecting Incorrect Location Embeddings Sepehr Sameni Simon Jenni Paolo Favaro ViT 34 4 0 10 Apr 2022
Unified Contrastive Learning in Image-Text-Label Space Jianwei Yang Chunyuan Li Pengchuan Zhang Bin Xiao Ce Liu Lu Yuan Jianfeng Gao VLM SSL 39 221 0 07 Apr 2022
Exploring Cross-Domain Pretrained Model for Hyperspectral Image Classification Hyungtae Lee Sungmin Eum H. Kwon 30 22 0 07 Apr 2022
MultiMAE: Multi-modal Multi-task Masked Autoencoders Roman Bachmann David Mizrahi Andrei Atanov Amir Zamir 47 265 0 04 Apr 2022
BatchFormerV2: Exploring Sample Relationships for Dense Representation Learning Zhi Hou Baosheng Yu Chaoyue Wang Yibing Zhan Dacheng Tao ViT 32 11 0 04 Apr 2022
Improving Vision Transformers by Revisiting High-frequency Components Jiawang Bai Liuliang Yuan Shutao Xia Shuicheng Yan Zhifeng Li Wei Liu ViT 16 90 0 03 Apr 2022
POS-BERT: Point Cloud One-Stage BERT Pre-Training Kexue Fu Peng Gao Shaolei Liu Renrui Zhang Yu Qiao Manning Wang 3DPC 30 18 0 03 Apr 2022
On the Importance of Asymmetry for Siamese Representation Learning Tianlin Li Haoqi Fan Yuandong Tian Daisuke Kihara Xinlei Chen SSL 30 51 0 01 Apr 2022
Self-distillation Augmented Masked Autoencoders for Histopathological Image Classification Yang Luo Zhineng Chen Shengtian Zhou Xieping Gao 31 1 0 31 Mar 2022
Dual Temperature Helps Contrastive Learning Without Many Negative Samples: Towards Understanding and Simplifying MoCo Chaoning Zhang Kang Zhang T. Pham Axi Niu Zhinan Qiao Chang D. Yoo In So Kweon 24 54 0 30 Mar 2022
mc-BEiT: Multi-choice Discretization for Image BERT Pre-training Xiaotong Li Yixiao Ge Kun Yi Zixuan Hu Ying Shan Ling-yu Duan 37 38 0 29 Mar 2022
Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap Yifei Wang Qi Zhang Yisen Wang Jiansheng Yang Zhouchen Lin 27 98 0 25 Mar 2022
Unsupervised Salient Object Detection with Spectral Cluster Voting Gyungin Shin Samuel Albanie Weidi Xie 24 65 0 23 Mar 2022
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training Zhan Tong Yibing Song Jue Wang Limin Wang ViT 143 1,129 0 23 Mar 2022
CP2: Copy-Paste Contrastive Pretraining for Semantic Segmentation Feng Wang Huiyu Wang Chen Wei Alan Yuille Wei Shen SSL VLM 25 35 0 22 Mar 2022
Domain Generalization by Mutual-Information Regularization with Pre-trained Models Junbum Cha Kyungjae Lee Sungrae Park Sanghyuk Chun OOD 26 131 0 21 Mar 2022
SimAN: Exploring Self-Supervised Representation Learning of Scene Text via Similarity-Aware Normalization Canjie Luo Lianwen Jin Jingdong Chen SSL AI4TS 22 29 0 20 Mar 2022
Object discovery and representation networks Olivier J. Hénaff Skanda Koppula Evan Shelhamer Daniel Zoran Andrew Jaegle Andrew Zisserman João Carreira Relja Arandjelović 44 87 0 16 Mar 2022
P-STMO: Pre-Trained Spatial Temporal Many-to-One Model for 3D Human Pose Estimation Wenkang Shan Zhenhua Liu Xinfeng Zhang Shanshe Wang Siwei Ma Wen Gao 3DH 34 121 0 15 Mar 2022
RecursiveMix: Mixed Learning with History Lingfeng Yang Xiang Li Borui Zhao Renjie Song Jian Yang VLM 29 18 0 14 Mar 2022
Scaling Up Your Kernels to 31x31: Revisiting Large Kernel Design in CNNs Xiaohan Ding Xinming Zhang Yi Zhou Jungong Han Guiguang Ding Jian Sun VLM 49 528 0 13 Mar 2022
Masked Visual Pre-training for Motor Control Tete Xiao Ilija Radosavovic Trevor Darrell Jitendra Malik SSL 34 242 0 11 Mar 2022
Backbone is All Your Need: A Simplified Architecture for Visual Object Tracking Boyu Chen Peixia Li Lei Bai Leixian Qiao Qiuhong Shen Bo-wen Li Weihao Gan Wei Wu Wanli Ouyang ViT VOT 22 183 0 10 Mar 2022
MVP: Multimodality-guided Visual Pre-training Longhui Wei Lingxi Xie Wen-gang Zhou Houqiang Li Qi Tian 28 106 0 10 Mar 2022
Multiscale Convolutional Transformer with Center Mask Pretraining for Hyperspectral Image Classification Sen Jia Yifan Wang ViT 41 13 0 09 Mar 2022
PASS: Part-Aware Self-Supervised Pre-Training for Person Re-Identification Kuan Zhu Haiyun Guo Tianyi Yan Yousong Zhu Jinqiao Wang Ming Tang 33 10 0 08 Mar 2022
Semi-Supervised Semantic Segmentation Using Unreliable Pseudo-Labels Yuchao Wang Haochen Wang Yujun Shen Jingjing Fei Wei Li Guoqiang Jin Liwei Wu Rui Zhao Xinyi Le UQCV 25 331 0 08 Mar 2022
DiT: Self-supervised Pre-training for Document Image Transformer Junlong Li Yiheng Xu Tengchao Lv Lei Cui Chaoxi Zhang Furu Wei ViT VLM 38 160 0 04 Mar 2022
A study on the distribution of social biases in self-supervised learning visual models Kirill Sirotkin Pablo Carballeira Marcos Escudero-Viñolo 22 18 0 03 Mar 2022
Audio Self-supervised Learning: A Survey Shuo Liu Adria Mallol-Ragolta Emilia Parada-Cabeleiro Kun Qian Xingshuo Jing Alexander Kathan Bin Hu Bjoern W. Schuller SSL 35 106 0 02 Mar 2022
Provable Stochastic Optimization for Global Contrastive Learning: Small Batch Does Not Harm Performance Zhuoning Yuan Yuexin Wu Zi-qi Qiu Xianzhi Du Lijun Zhang Denny Zhou Tianbao Yang 34 26 0 24 Feb 2022
Indiscriminate Poisoning Attacks on Unsupervised Contrastive Learning Hao He Kaiwen Zha Dina Katabi AAML 34 32 0 22 Feb 2022
GroupViT: Semantic Segmentation Emerges from Text Supervision Jiarui Xu Shalini De Mello Sifei Liu Wonmin Byeon Thomas Breuel Jan Kautz Xinyu Wang ViT VLM 192 501 0 22 Feb 2022
Vision-Language Pre-Training with Triple Contrastive Learning Jinyu Yang Jiali Duan Son N. Tran Yi Xu Sampath Chanda Liqun Chen Belinda Zeng Trishul Chilimbi Junzhou Huang VLM 34 289 0 21 Feb 2022
Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision Priya Goyal Quentin Duval Isaac Seessel Mathilde Caron Ishan Misra Levent Sagun Armand Joulin Piotr Bojanowski VLM SSL 26 110 0 16 Feb 2022
Meta Knowledge Distillation Jihao Liu Boxiao Liu Hongsheng Li Yu Liu 18 25 0 16 Feb 2022
Distillation with Contrast is All You Need for Self-Supervised Point Cloud Representation Learning Kexue Fu Peng Gao Renrui Zhang Hongsheng Li Yu Qiao Manning Wang SSL 3DPC 28 23 0 09 Feb 2022
Self-supervised Contrastive Learning for Cross-domain Hyperspectral Image Representation Hyungtae Lee H. Kwon SSL 22 17 0 08 Feb 2022