Emerging Property of Masked Token for Effective Pre-training

12 April 2024

Hyesong Choi

Hyejin Park

Dongbo Min

Papers citing "Emerging Property of Masked Token for Effective Pre-training"

21 / 21 papers shown

Title
iConFormer: Dynamic Parameter-Efficient Tuning with Input-Conditioned Adaptation Hayeon Jo Hyesong Choi Minhee Cho Dongbo Min 73 1 0 04 Sep 2024
Collaborative Learning for Enhanced Unsupervised Domain Adaptation Minhee Cho Hyesong Choi Hayeon Jo Dongbo Min 120 1 0 04 Sep 2024
A Unified View of Masked Image Modeling Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei VLM 83 35 0 19 Oct 2022
BEiT v2: Masked Image Modeling with Vector-Quantized Visual Tokenizers Zhiliang Peng Li Dong Hangbo Bao QiXiang Ye Furu Wei 51 316 0 12 Aug 2022
MILAN: Masked Image Pretraining on Language Assisted Representation Zejiang Hou Fei Sun Yen-kuang Chen Yuan Xie S. Kung ViT 48 68 0 11 Aug 2022
A Survey on Masked Autoencoder for Self-supervised Learning in Vision and Beyond Chaoning Zhang Chenshuang Zhang Junha Song John Seon Keun Yi Kang Zhang In So Kweon SSL 70 76 0 30 Jul 2022
Bootstrapped Masked Autoencoders for Vision BERT Pretraining Xiaoyi Dong Jianmin Bao Ting Zhang Dongdong Chen Weiming Zhang Lu Yuan Dong Chen Fang Wen Nenghai Yu 59 76 0 14 Jul 2022
Object-wise Masked Autoencoders for Fast Pre-training Jiantao Wu Shentong Mo ViT OCL 55 15 0 28 May 2022
Green Hierarchical Vision Transformer for Masked Image Modeling Lang Huang Shan You Mingkai Zheng Fei Wang Chao Qian T. Yamasaki 87 69 0 26 May 2022
Masked Image Modeling with Denoising Contrast Kun Yi Yixiao Ge Xiaotong Li Shusheng Yang Dian Li Jianping Wu Ying Shan Xiaohu Qie VLM 37 52 0 19 May 2022
mc-BEiT: Multi-choice Discretization for Image BERT Pre-training Xiaotong Li Yixiao Ge Kun Yi Zixuan Hu Ying Shan Ling-yu Duan 59 39 0 29 Mar 2022
Should You Mask 15% in Masked Language Modeling? Alexander Wettig Tianyu Gao Zexuan Zhong Danqi Chen CVBM 51 163 0 16 Feb 2022
Swin Transformer V2: Scaling Up Capacity and Resolution Ze Liu Han Hu Yutong Lin Zhuliang Yao Zhenda Xie ... Yue Cao Zheng Zhang Li Dong Furu Wei B. Guo ViT 182 1,783 0 18 Nov 2021
iBOT: Image BERT Pre-Training with Online Tokenizer Jinghao Zhou Chen Wei Huiyu Wang Wei Shen Cihang Xie Alan Yuille Tao Kong 59 722 0 15 Nov 2021
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Elad Ben-Zaken Shauli Ravfogel Yoav Goldberg 145 1,191 0 18 Jun 2021
BEiT: BERT Pre-Training of Image Transformers Hangbo Bao Li Dong Songhao Piao Furu Wei ViT 175 2,790 0 15 Jun 2021
EarlyBERT: Efficient BERT Training via Early-bird Lottery Tickets Xiaohan Chen Yu Cheng Shuohang Wang Zhe Gan Zhangyang Wang Jingjing Liu 65 100 0 31 Dec 2020
MPNet: Masked and Permuted Pre-training for Language Understanding Kaitao Song Xu Tan Tao Qin Jianfeng Lu Tie-Yan Liu 94 1,105 0 20 Apr 2020
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 272 6,420 0 26 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 415 24,160 0 26 Jul 2019
MASS: Masked Sequence to Sequence Pre-training for Language Generation Kaitao Song Xu Tan Tao Qin Jianfeng Lu Tie-Yan Liu 92 962 0 07 May 2019