InforMask: Unsupervised Informative Masking for Language Model Pretraining

21 October 2022

Papers citing "InforMask: Unsupervised Informative Masking for Language Model Pretraining"

14 / 14 papers shown

Title
EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning Ashish Seth Ramaneswaran Selvakumar S. Sakshi Sonal Kumar Sreyan Ghosh Dinesh Manocha 24 0 0 17 Oct 2024
Unlocking Efficiency: Adaptive Masking for Gene Transformer Models Soumyadeep Roy S. Sural Niloy Ganguly MedIm 37 0 0 13 Aug 2024
The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMs Paul Youssef Jorg Schlotterer Christin Seifert KELM 26 2 0 02 Feb 2024
Lil-Bevo: Explorations of Strategies for Training Language Models in More Humanlike Ways Venkata S Govindarajan Juan Diego Rodriguez Kaj Bostrom Kyle Mahowald 15 1 0 26 Oct 2023
Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models Paul Youssef Osman Alperen Koracs Meijie Li Jorg Schlotterer Christin Seifert KELM 24 16 0 25 Oct 2023
GeneMask: Fast Pretraining of Gene Sequences to Enable Few-Shot Learning Soumyadeep Roy Jonas Wallat Sowmya S. Sundaram Wolfgang Nejdl Niloy Ganguly 27 3 0 29 Jul 2023
Self-Evolution Learning for Discriminative Language Model Pretraining Qihuang Zhong Liang Ding Juhua Liu Bo Du Dacheng Tao 34 12 0 24 May 2023
Challenging Decoder helps in Masked Auto-Encoder Pre-training for Dense Passage Retrieval Zehan Li Yanzhao Zhang Dingkun Long Pengjun Xie RALM 33 3 0 22 May 2023
Unsupervised Improvement of Factual Knowledge in Language Models Nafis Sadeq Byungkyu Kang Prarit Lamba Julian McAuley KELM 19 6 0 04 Apr 2023
Toward Efficient Language Model Pretraining and Downstream Adaptation via Self-Evolution: A Case Study on SuperGLUE Qihuang Zhong Liang Ding Yibing Zhan Yu Qiao Yonggang Wen ... Yixin Chen Xinbo Gao Chun Miao Xiaoou Tang Dacheng Tao VLM ELM 55 34 0 04 Dec 2022
K-BERT: Enabling Language Representation with Knowledge Graph Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Qi Ju Haotang Deng Ping Wang 231 778 0 17 Sep 2019
Knowledge Enhanced Contextual Word Representations Matthew E. Peters Mark Neumann IV RobertL.Logan Roy Schwartz Vidur Joshi Sameer Singh Noah A. Smith 231 656 0 09 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 415 2,586 0 03 Sep 2019
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 297 6,959 0 20 Apr 2018