v1v2 (latest)

On the Influence of Masking Policies in Intermediate Pre-training

18 April 2021

Sinong Wang

Hao Ma

Xiang Ren

Madian Khabsa

ArXiv (abs)PDF HTML

Papers citing "On the Influence of Masking Policies in Intermediate Pre-training"

33 / 33 papers shown

Title
Studying Strategically: Learning to Mask for Closed-book QA Qinyuan Ye Belinda Z. Li Sinong Wang Benjamin Bolte Hao Ma Wen-tau Yih Xiang Ren Madian Khabsa OffRL 79 12 0 31 Dec 2020
Neural Mask Generator: Learning to Generate Adaptive Word Maskings for Language Model Adaptation Minki Kang Moonsu Han Sung Ju Hwang OOD 56 18 0 06 Oct 2020
PMI-Masking: Principled masking of correlated spans Yoav Levine Barak Lenz Opher Lieber Omri Abend Kevin Leyton-Brown Moshe Tennenholtz Y. Shoham 58 73 0 05 Oct 2020
Variance-reduced Language Pretraining via a Mask Proposal Network Liang Chen SSL 48 8 0 12 Aug 2020
UnifiedQA: Crossing Format Boundaries With a Single QA System Daniel Khashabi Sewon Min Tushar Khot Ashish Sabharwal Oyvind Tafjord Peter Clark Hannaneh Hajishirzi 153 742 0 02 May 2020
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work? Yada Pruksachatkun Jason Phang Haokun Liu Phu Mon Htut Xiaoyi Zhang Richard Yuanzhe Pang Clara Vania Katharina Kann Samuel R. Bowman CLL LRM 72 197 0 01 May 2020
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Suchin Gururangan Ana Marasović Swabha Swayamdipta Kyle Lo Iz Beltagy Doug Downey Noah A. Smith VLM AI4CE CLL 167 2,440 0 23 Apr 2020
Train No Evil: Selective Masking for Task-Guided Pre-Training Yuxian Gu Zhengyan Zhang Xiaozhi Wang Zhiyuan Liu Maosong Sun 136 59 0 21 Apr 2020
Extractive Summarization as Text Matching Ming Zhong Pengfei Liu Yiran Chen Danqing Wang Xipeng Qiu Xuanjing Huang 150 462 0 19 Apr 2020
How Much Knowledge Can You Pack Into the Parameters of a Language Model? Adam Roberts Colin Raffel Noam M. Shazeer KELM 138 897 0 10 Feb 2020
REALM: Retrieval-Augmented Language Model Pre-Training Kelvin Guu Kenton Lee Zora Tung Panupong Pasupat Ming-Wei Chang RALM 147 2,121 0 10 Feb 2020
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization Jingqing Zhang Yao-Min Zhao Mohammad Saleh Peter J. Liu RALM 3DGS 299 2,056 0 18 Dec 2019
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 266 10,880 0 29 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 506 20,376 0 23 Oct 2019
Generalized Inner Loop Meta-Learning Jaya Kumar Alageshan Brandon Amos A. Verma Phu Mon Htut Artem Molchanov Franziska Meier Douwe Kiela Kyunghyun Cho Soumith Chintala AI4CE 95 160 0 03 Oct 2019
WIQA: A dataset for "What if..." reasoning over procedural text Niket Tandon Bhavana Dalvi Keisuke Sakaguchi Antoine Bosselut Peter Clark 72 102 0 10 Sep 2019
QuaRTz: An Open-Domain Dataset of Qualitative Relationship Questions Oyvind Tafjord Matt Gardner Kevin Lin Peter Clark 61 111 0 08 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 587 2,680 0 03 Sep 2019
Reasoning Over Paragraph Effects in Situations Kevin Lin Oyvind Tafjord Peter Clark Matt Gardner 90 115 0 16 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 700 24,572 0 26 Jul 2019
SpanBERT: Improving Pre-training by Representing and Predicting Spans Mandar Joshi Danqi Chen Yinhan Liu Daniel S. Weld Luke Zettlemoyer Omer Levy 174 1,970 0 24 Jul 2019
fairseq: A Fast, Extensible Toolkit for Sequence Modeling Myle Ott Sergey Edunov Alexei Baevski Angela Fan Sam Gross Nathan Ng David Grangier Michael Auli VLM FaML 132 3,159 0 01 Apr 2019
Wizard of Wikipedia: Knowledge-Powered Conversational agents Emily Dinan Stephen Roller Kurt Shuster Angela Fan Michael Auli Jason Weston RALM KELM 150 950 0 03 Nov 2018
How to train your MAML Antreas Antoniou Harrison Edwards Amos Storkey 74 778 0 22 Oct 2018
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.8K 95,324 0 11 Oct 2018
Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization Shashi Narayan Shay B. Cohen Mirella Lapata AILaw 161 1,686 0 27 Aug 2018
A Simple Method for Commonsense Reasoning Trieu H. Trinh Quoc V. Le LRM ReLM 102 434 0 07 Jun 2018
Zero-Shot Relation Extraction via Reading Comprehension Omer Levy Minjoon Seo Eunsol Choi Luke Zettlemoyer ReLM 86 699 0 13 Jun 2017
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 245 2,692 0 09 May 2017
Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks Chelsea Finn Pieter Abbeel Sergey Levine OOD 835 11,961 0 09 Mar 2017
Categorical Reparameterization with Gumbel-Softmax Eric Jang S. Gu Ben Poole BDL 367 5,390 0 03 Nov 2016
Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books Yukun Zhu Ryan Kiros R. Zemel Ruslan Salakhutdinov R. Urtasun Antonio Torralba Sanja Fidler 142 2,555 0 22 Jun 2015
Teaching Machines to Read and Comprehend Karl Moritz Hermann Tomás Kociský Edward Grefenstette L. Espeholt W. Kay Mustafa Suleyman Phil Blunsom 355 3,555 0 10 Jun 2015