Train No Evil: Selective Masking for Task-Guided Pre-Training

Train No Evil: Selective Masking for Task-Guided Pre-Training

21 April 2020

Zhengyan Zhang

Xiaozhi Wang

Maosong Sun

Papers citing "Train No Evil: Selective Masking for Task-Guided Pre-Training"

17 / 17 papers shown

Title
The Efficiency of Pre-training with Objective Masking in Pseudo Labeling for Semi-Supervised Text Classification Arezoo Hatefi Xuan-Son Vu Monowar Bhuyan Frank Drewes VLM 35 0 0 10 May 2025
CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Search Kaixin Wu Yixin Ji Ziyang Chen Qiang Wang Cunxiang Wang ... Jia Xu Zhongyi Liu Jinjie Gu Yuan Zhou Linjian Mo KELM CLL 92 0 0 02 Dec 2024
How Useful is Continued Pre-Training for Generative Unsupervised Domain Adaptation? Rheeya Uppaal Yixuan Li Junjie Hu 37 4 0 31 Jan 2024
An Anchor Learning Approach for Citation Field Learning Zilin Yuan Borun Chen Yimeng Dai Hai-Tao Zheng Hai-Tao Zheng Rui Zhang 38 0 0 07 Sep 2023
Do not Mask Randomly: Effective Domain-adaptive Pre-training by Masking In-domain Keywords Shahriar Golchin Mihai Surdeanu N. Tavabi A. Kiapour 18 4 0 14 Jul 2023
Difference-Masking: Choosing What to Mask in Continued Pretraining Alex Wilf Syeda Nahida Akter Leena Mathur Paul Pu Liang Sheryl Mathew Mengrou Shou Eric Nyberg Louis-Philippe Morency CLL SSL 32 4 0 23 May 2023
Teaching the Pre-trained Model to Generate Simple Texts for Text Simplification Renliang Sun Wei-ping Xu Xiaojun Wan CLL 21 17 0 21 May 2023
APOLLO: A Simple Approach for Adaptive Pretraining of Language Models for Logical Reasoning Soumya Sanyal Yichong Xu Shuohang Wang Ziyi Yang Reid Pryzant W. Yu Chenguang Zhu Xiang Ren ReLM LRM 35 8 0 19 Dec 2022
Using Selective Masking as a Bridge between Pre-training and Fine-tuning Tanish Lad Himanshu Maheshwari Shreyas Kottukkal R. Mamidi 24 3 0 24 Nov 2022
Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base Cunxiang Wang Fuli Luo Yanyang Li Runxin Xu Fei Huang Yue Zhang KELM 33 2 0 17 Apr 2022
A Survey on Dropout Methods and Experimental Verification in Recommendation Yong Li Weizhi Ma C. L. Philip Chen Hao Fei Yiqun Liu Shaoping Ma Yue Yang 33 9 0 05 Apr 2022
Fortunately, Discourse Markers Can Enhance Language Models for Sentiment Analysis L. Ein-Dor Ilya Shnayderman Artem Spector Lena Dankin R. Aharonov Noam Slonim 28 8 0 06 Jan 2022
MoCA: Incorporating Multi-stage Domain Pretraining and Cross-guided Multimodal Attention for Textbook Question Answering Fangzhi Xu Qika Lin Xiaozhong Liu Lingling Zhang Tianzhe Zhao Qianyi Chai Yudai Pan 14 2 0 06 Dec 2021
Pre-Trained Models: Past, Present and Future Xu Han Zhengyan Zhang Ning Ding Yuxian Gu Xiao Liu ... Jie Tang Ji-Rong Wen Jinhui Yuan Wayne Xin Zhao Jun Zhu AIFin MQ AI4MH 40 815 0 14 Jun 2021
CLEVE: Contrastive Pre-training for Event Extraction Ziqi Wang Xiaozhi Wang Xu Han Yankai Lin Lei Hou Zhiyuan Liu Peng Li Juan-Zi Li Jie Zhou 37 116 0 30 May 2021
Studying Strategically: Learning to Mask for Closed-book QA Qinyuan Ye Belinda Z. Li Sinong Wang Benjamin Bolte Hao Ma Wen-tau Yih Xiang Ren Madian Khabsa OffRL 24 11 0 31 Dec 2020
Improving Low Compute Language Modeling with In-Domain Embedding Initialisation Charles F Welch Rada Mihalcea Jonathan K. Kummerfeld AI4CE 11 4 0 29 Sep 2020