Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality

20 May 2022

Xiang Li

Wenhai Wang

Lingfeng Yang

Jian Yang

ArXiv PDF HTML

Papers citing "Uniform Masking: Enabling MAE Pre-training for Pyramid-based Vision Transformers with Locality"

50 / 56 papers shown

Title
Reinforcement Learning meets Masked Video Modeling : Trajectory-Guided Adaptive Token Selection Ayush K. Rai Kyle Min Tarun Krishna Feiyan Hu Alan F. Smeaton Noel E. O'Connor VGen 31 0 0 13 May 2025
Wavelet-Driven Masked Image Modeling: A Path to Efficient Visual Representation Wenzhao Xiang Chang Liu Hongyang Yu Xilin Chen 33 0 0 02 Mar 2025
Intelligent Anomaly Detection for Lane Rendering Using Transformer with Self-Supervised Pre-Training and Customized Fine-Tuning Yongqi Dong Xingmin Lu Ruohan Li Wei Song B. Arem Haneen Farah ViT 107 1 0 21 Feb 2025
Pre-training a Density-Aware Pose Transformer for Robust LiDAR-based 3D Human Pose Estimation Xiaoqi An Lin Zhao Chen Gong Jun Yu Li Jian Yang 3DH 80 0 0 18 Dec 2024
RingMo-Aerial: An Aerial Remote Sensing Foundation Model With A Affine Transformation Contrastive Learning Wenhui Diao Haichen Yu Kaiyue Kang Tong Ling Di Liu ... Hanbo Bi Libo Ren Xuexue Li Yongqiang Mao Xian Sun 34 1 0 20 Sep 2024
Masked Angle-Aware Autoencoder for Remote Sensing Images Zhihao Li B. Hou Siteng Ma Zitong Wu Xianpeng Guo Bo Ren Licheng Jiao 46 11 0 04 Aug 2024
Improving Representation of High-frequency Components for Medical Visual Foundation Models Yuetan Chu Yilan Zhang Zhongyi Han Changchun Yang Longxi Zhou Gongning Luo Chao Huang Xin Gao MedIm 47 1 0 19 Jul 2024
ColorMAE: Exploring data-independent masking strategies in Masked AutoEncoders Carlos Hinojosa Shuming Liu Guohao Li 26 2 0 17 Jul 2024
Pseudo Labelling for Enhanced Masked Autoencoders S. Nandam Sara Atito Zhenhua Feng Josef Kittler Muhammad Awais 64 1 0 25 Jun 2024
Cacophony: An Improved Contrastive Audio-Text Model Ge Zhu Jordan Darefsky Zhiyao Duan AuLLM 46 11 0 10 Feb 2024
Masked Modeling for Self-supervised Representation Learning on Vision and Beyond Siyuan Li Luyuan Zhang Zedong Wang Di Wu Lirong Wu ... Jun-Xiong Xia Cheng Tan Yang Liu Baigui Sun Stan Z. Li SSL 36 14 0 31 Dec 2023
Bootstrap Masked Visual Modeling via Hard Patches Mining Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tiancai Wang Xiangyu Zhang Zhaoxiang Zhang 39 5 0 21 Dec 2023
M-BEV: Masked BEV Perception for Robust Autonomous Driving Siran Chen Yue Ma Yu Qiao Yali Wang 33 8 0 19 Dec 2023
LMD: Faster Image Reconstruction with Latent Masking Diffusion Zhiyuan Ma Zhihuan Yu Jianjun Li Bowen Zhou DiffM 24 8 0 13 Dec 2023
MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness Xiaoyun Xu Shujian Yu Jingzheng Wu S. Picek AAML 35 0 0 08 Dec 2023
Local Masking Meets Progressive Freezing: Crafting Efficient Vision Transformers for Self-Supervised Learning Utku Mert Topcuoglu Erdem Akagündüz 43 1 0 02 Dec 2023
E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer Jacob Zhiyuan Fang Skyler Zheng Vasu Sharma Robinson Piramuthu VLM 38 0 0 28 Nov 2023
Dual-channel Prototype Network for few-shot Classification of Pathological Images Hao Quan Xinjia Li Dayu Hu Tianhang Nan Xiaoyu Cui 19 0 0 14 Nov 2023
Limited Data, Unlimited Potential: A Study on ViTs Augmented by Masked Autoencoders Srijan Das Tanmay Jain Dominick Reilly P. Balaji Soumyajit Karmakar Shyam Marjit Xiang Li Abhijit Das Michael S. Ryoo 39 16 0 31 Oct 2023
Bridging The Gaps Between Token Pruning and Full Pre-training via Masked Fine-tuning Fengyuan Shi Limin Wang ViT 38 0 0 26 Oct 2023
AMLP:Adaptive Masking Lesion Patches for Self-supervised Medical Image Segmentation Xiang-Fei Wang Ruizhi Wang Jie Zhou Thomas Lukasiewicz Zhenghua Xu 22 0 0 08 Sep 2023
DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positions Haochen Wang Junsong Fan Yuxi Wang Kaiyou Song Tong Wang Zhaoxiang Zhang 29 19 0 07 Sep 2023
CL-MAE: Curriculum-Learned Masked Autoencoders Neelu Madan Nicolae-Cătălin Ristea Kamal Nasrollahi T. Moeslund Radu Tudor Ionescu 19 10 0 31 Aug 2023
Masked Feature Modelling: Feature Masking for the Unsupervised Pre-training of a Graph Attention Network Block for Bottom-up Video Event Recognition Dimitrios Daskalakis Nikolaos Gkalelis Vasileios Mezaris 36 0 0 24 Aug 2023
SRMAE: Masked Image Modeling for Scale-Invariant Deep Representations Zhiming Wang Lin Gu Feng Lu 28 0 0 17 Aug 2023
MixReorg: Cross-Modal Mixed Patch Reorganization is a Good Mask Learner for Open-World Semantic Segmentation Kaixin Cai Pengzhen Ren Yi Zhu Hang Xu Jian-zhuo Liu Changlin Li Guangrun Wang Xiaodan Liang VLM 29 14 0 09 Aug 2023
Enhanced Masked Image Modeling for Analysis of Dental Panoramic Radiographs A. Almalki Longin Jan Latecki MedIm 15 4 0 18 Jun 2023
UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks Yanan Sun Zi-Qi Zhong Qi Fan Chi-Keung Tang Yu-Wing Tai VLM 30 4 0 07 Jun 2023
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles Chaitanya K. Ryali Yuan-Ting Hu Daniel Bolya Chen Wei Haoqi Fan ... Omid Poursaeed Judy Hoffman Jitendra Malik Yanghao Li Christoph Feichtenhofer 3DH 45 159 0 01 Jun 2023
Image as First-Order Norm+Linear Autoregression: Unveiling Mathematical Invariance Yinpeng Chen Xiyang Dai Dongdong Chen Mengchen Liu Lu Yuan Zicheng Liu Youzuo Lin 26 2 0 25 May 2023
A Cookbook of Self-Supervised Learning Randall Balestriero Mark Ibrahim Vlad Sobal Ari S. Morcos Shashank Shekhar ... Pierre Fernandez Amir Bar Hamed Pirsiavash Yann LeCun Micah Goldblum SyDa FedML SSL 44 273 0 24 Apr 2023
TransFlow: Transformer as Flow Learner Yawen Lu Qifan Wang Siqi Ma Tong Geng Victor Y. Chen Huaijin Chen Dongfang Liu ViT 32 45 0 23 Apr 2023
Hard Patches Mining for Masked Image Modeling Haochen Wang Kaiyou Song Junsong Fan Yuxi Wang Jin Xie Zhaoxiang Zhang 34 59 0 12 Apr 2023
Mask and Restore: Blind Backdoor Defense at Test Time with Masked Autoencoder Tao Sun Lu Pang Chao Chen Haibin Ling AAML 43 9 0 27 Mar 2023
Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimicking Peng Gao Renrui Zhang Rongyao Fang Ziyi Lin Hongyang Li Hongsheng Li Qiao Yu 19 18 0 09 Mar 2023
Masked Image Modeling with Local Multi-Scale Reconstruction Haoqing Wang Yehui Tang Yunhe Wang Jianyuan Guo Zhiwei Deng Kai Han 61 46 0 09 Mar 2023
Remote Sensing Scene Classification with Masked Image Modeling (MIM) Liya Wang A. Tien 35 3 0 28 Feb 2023
MPS-AMS: Masked Patches Selection and Adaptive Masking Strategy Based Self-Supervised Medical Image Segmentation Xiang-Fei Wang Ruizhi Wang Biao Tian Jiaojiao Zhang Shuo Zhang Junyang Chen Thomas Lukasiewicz Zhenghua Xu 22 1 0 27 Feb 2023
Aerial Image Object Detection With Vision Transformer Detector (ViTDet) Liya Wang A. Tien 44 7 0 28 Jan 2023
Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting Problem Peiwang Tang Xianchao Zhang AI4TS 26 3 0 04 Jan 2023
Disjoint Masking with Joint Distillation for Efficient Masked Image Modeling Xin Ma Chang-Shu Liu Chunyu Xie Long Ye Yafeng Deng Xiang Ji 25 9 0 31 Dec 2022
FastMIM: Expediting Masked Image Modeling Pre-training for Vision Jianyuan Guo Kai Han Han Wu Yehui Tang Yunhe Wang Chang Xu 33 9 0 13 Dec 2022
Masked Contrastive Representation Learning Yuan Yao Nandakishor Desai M. Palaniswami SSL 22 8 0 11 Nov 2022
Rethinking Hierarchies in Pre-trained Plain Vision Transformer Yufei Xu Jing Zhang Qiming Zhang Dacheng Tao 18 1 0 03 Nov 2022
Self-Supervised Learning with Masked Image Modeling for Teeth Numbering, Detection of Dental Restorations, and Instance Segmentation in Dental Panoramic Radiographs A. Almalki Longin Jan Latecki MedIm 22 14 0 20 Oct 2022
MTSMAE: Masked Autoencoders for Multivariate Time-Series Forecasting Peiwang Tang Xianchao Zhang AI4TS 27 12 0 04 Oct 2022
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond Chaoning Zhang Chenshuang Zhang Junha Song John Seon Keun Yi Kang Zhang In So Kweon SSL 57 71 0 30 Jul 2022
MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformers Jihao Liu Xin Huang Jinliang Zheng Yu Liu Hongsheng Li 33 53 0 26 May 2022
Context Autoencoder for Self-Supervised Representation Learning Xiaokang Chen Mingyu Ding Xiaodi Wang Ying Xin Shentong Mo Yunhao Wang Shumin Han Ping Luo Gang Zeng Jingdong Wang SSL 45 385 0 07 Feb 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021