Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality

20 May 2022

Xiang Li

Wenhai Wang

Lingfeng Yang

Jian Yang

ArXiv PDF HTML

Papers citing "Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality"

50 / 56 papers shown

Title
Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection Ayush K. Rai Kyle Min Tarun Krishna Feiyan Hu Alan F. Smeaton Noel E. O'Connor VGen 31 0 0 13 May 2025
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation Wenzhao Xiang Chang Liu Hongyang Yu Xilin Chen 36 0 0 02 Mar 2025
Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning Yongqi Dong Xingmin Lu Ruohan Li Wei Song B. Arem Haneen Farah ViT 110 1 0 21 Feb 2025
Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation Xiaoqi An Lin Zhao Chen Gong Jun Yu Li Jian Yang 3DH 80 0 0 18 Dec 2024
RingMo-Aerial: An Aerial Remote Sensing Foundation Model With A Affine Transformation Contrastive Learning Wenhui Diao Haichen Yu Kaiyue Kang Tong Ling Di Liu ... Hanbo Bi Libo Ren Xuexue Li Yongqiang Mao Xian Sun 37 1 0 20 Sep 2024
Masked Angle-Aware Autoencoder for Remote Sensing Images Zhihao Li B. Hou Siteng Ma Zitong Wu Xianpeng Guo Bo Ren Licheng Jiao 49 11 0 04 Aug 2024
Improving Representation of High-frequency Components for Medical Visual Foundation Models Yuetan Chu Yilan Zhang Zhongyi Han Changchun Yang Longxi Zhou Gongning Luo Chao Huang Xin Gao MedIm 47 1 0 19 Jul 2024
ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders Carlos Hinojosa Shuming Liu Guohao Li 26 0 0 17 Jul 2024
Pseudo Labelling for Enhanced Masked Autoencoders S. Nandam Sara Atito Zhenhua Feng Josef Kittler Muhammad Awais 67 1 0 25 Jun 2024
Cacophony: An Improved Contrastive Audio-Text Model Ge Zhu Jordan Darefsky Zhiyao Duan AuLLM 46 11 0 10 Feb 2024
Masked Modeling for Self-supervised Representation Learning on Vision and Beyond Siyuan Li Luyuan Zhang Zedong Wang Di Wu Lirong Wu ... Jun Xia Cheng Tan Yang Liu Baigui Sun Stan Z. Li SSL 39 14 0 31 Dec 2023
Bootstrap Masked Visual Modeling via Hard Patches Mining Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tiancai Wang Xiangyu Zhang Zhaoxiang Zhang 42 5 0 21 Dec 2023
M-BEV: Masked BEV Perception for Robust Autonomous Driving Siran Chen Yue Ma Yu Qiao Yali Wang 36 8 0 19 Dec 2023
LMD: Faster Image Reconstruction with Latent Masking Diffusion Zhiyuan Ma Zhihuan Yu Jianjun Li Bowen Zhou DiffM 24 8 0 13 Dec 2023
MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness Xiaoyun Xu Shujian Yu Jingzheng Wu S. Picek AAML 35 0 0 08 Dec 2023
Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning Utku Mert Topcuoglu Erdem Akagündüz 43 1 0 02 Dec 2023
E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer Jacob Zhiyuan Fang Skyler Zheng Vasu Sharma Robinson Piramuthu VLM 38 0 0 28 Nov 2023
Dual-channel Prototype Network for few-shot Classification of Pathological Images Hao Quan Xinjia Li Dayu Hu Tianhang Nan Xiaoyu Cui 19 0 0 14 Nov 2023
Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders Srijan Das Tanmay Jain Dominick Reilly P. Balaji Soumyajit Karmakar Shyam Marjit Xiang Li Abhijit Das Michael S. Ryoo 39 16 0 31 Oct 2023
Bridging The Gaps Between Token Pruning and Full Pre-training via Masked Fine-tuning Fengyuan Shi Limin Wang ViT 38 0 0 26 Oct 2023
AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation Xiang-Fei Wang Ruizhi Wang Jie Zhou Thomas Lukasiewicz Zhenghua Xu 24 0 0 08 Sep 2023
DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tong Wang Zhaoxiang Zhang 29 19 0 07 Sep 2023
CL-MAE: Curriculum-Learned Masked Autoencoders Neelu Madan Nicolae-Cătălin Ristea Kamal Nasrollahi T. Moeslund Radu Tudor Ionescu 19 10 0 31 Aug 2023
Masked Feature Modelling: Feature Masking for the Unsupervised Pre-training of a Graph Attention Network Block for Bottom-up Video Event Recognition Dimitrios Daskalakis Nikolaos Gkalelis Vasileios Mezaris 36 0 0 24 Aug 2023
SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations Zhiming Wang Lin Gu Feng Lu 28 0 0 17 Aug 2023
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation Kaixin Cai Pengzhen Ren Yi Zhu Hang Xu Jian-zhuo Liu Changlin Li Guangrun Wang Xiaodan Liang VLM 29 14 0 09 Aug 2023
Enhanced Masked Image Modeling for Analysis of Dental Panoramic Radiographs A. Almalki Longin Jan Latecki MedIm 20 4 0 18 Jun 2023
UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks Yanan Sun Zi-Qi Zhong Qi Fan Chi-Keung Tang Yu-Wing Tai VLM 33 4 0 07 Jun 2023
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles Chaitanya K. Ryali Yuan-Ting Hu Daniel Bolya Chen Wei Haoqi Fan ... Omid Poursaeed Judy Hoffman Jitendra Malik Yanghao Li Christoph Feichtenhofer 3DH 45 160 0 01 Jun 2023
Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Lu Yuan Zicheng Liu Youzuo Lin 26 2 0 25 May 2023
A Cookbook of Self-Supervised Learning Randall Balestriero Mark Ibrahim Vlad Sobal Ari S. Morcos Shashank Shekhar ... Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum SyDa FedML SSL 47 274 0 24 Apr 2023
TransFlow: Transformer as Flow Learner Yawen Lu Qifan Wang Siqi Ma Tong Geng Victor Y. Chen Huaijin Chen Dongfang Liu ViT 35 45 0 23 Apr 2023
Hard Patches Mining for Masked Image Modeling Haochen Wang Kaiyou Song Junsong Fan Yuxi Wang Jin Xie Zhaoxiang Zhang 37 59 0 12 Apr 2023
Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder Tao Sun Lu Pang Chao Chen Haibin Ling AAML 43 9 0 27 Mar 2023
Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking Peng Gao Renrui Zhang Rongyao Fang Ziyi Lin Hongyang Li Hongsheng Li Qiao Yu 24 18 0 09 Mar 2023
Masked Image Modeling with Local Multi-Scale Reconstruction Haoqing Wang Yehui Tang Yunhe Wang Jianyuan Guo Zhiwei Deng Kai Han 61 46 0 09 Mar 2023
Remote Sensing Scene Classification with Masked Image Modeling (MIM) Liya Wang A. Tien 35 3 0 28 Feb 2023
MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation Xiang-Fei Wang Ruizhi Wang Biao Tian Jiaojiao Zhang Shuo Zhang Junyang Chen Thomas Lukasiewicz Zhenghua Xu 30 1 0 27 Feb 2023
Aerial Image Object Detection With Vision Transformer Detector (ViTDet) Liya Wang A. Tien 46 7 0 28 Jan 2023
Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problem Peiwang Tang Xianchao Zhang AI4TS 31 3 0 04 Jan 2023
Disjoint Masking with Joint Distillation for Efficient Masked Image Modeling Xin Ma Chang-Shu Liu Chunyu Xie Long Ye Yafeng Deng Xiang Ji 25 9 0 31 Dec 2022
FastMIM: Expediting Masked Image Modeling Pre-training for Vision Jianyuan Guo Kai Han Han Wu Yehui Tang Yunhe Wang Chang Xu 33 9 0 13 Dec 2022
Masked Contrastive Representation Learning Yuan Yao Nandakishor Desai M. Palaniswami SSL 22 8 0 11 Nov 2022
Rethinking Hierarchies in Pre-trained Plain Vision Transformer Yufei Xu Jing Zhang Qiming Zhang Dacheng Tao 18 1 0 03 Nov 2022
Self-Supervised Learning with Masked Image Modeling for Teeth Numbering, Detection of Dental Restorations, and Instance Segmentation in Dental Panoramic Radiographs A. Almalki Longin Jan Latecki MedIm 22 14 0 20 Oct 2022
MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting Peiwang Tang Xianchao Zhang AI4TS 35 12 0 04 Oct 2022
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond Chaoning Zhang Chenshuang Zhang Junha Song John Seon Keun Yi Kang Zhang In So Kweon SSL 57 71 0 30 Jul 2022
MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers Jihao Liu Xin Huang Jinliang Zheng Yu Liu Hongsheng Li 33 53 0 26 May 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 45 385 0 07 Feb 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 308 7,443 0 11 Nov 2021