Don't Stop Pretraining: Adapt Language Models to Domains and Tasks

23 April 2020

Kyle Lo

Papers citing "Don't Stop Pretraining: Adapt Language Models to Domains and Tasks"

50 / 522 papers shown

Title
Detoxifying Language Models Risks Marginalizing Minority Voices Albert Xu Eshaan Pathak Eric Wallace Suchin Gururangan Maarten Sap Dan Klein 24 123 0 13 Apr 2021
Semantic maps and metrics for science Semantic maps and metrics for science using deep transformer encoders Brendan Chambers James A. Evans MedIm 13 0 0 13 Apr 2021
CURIE: An Iterative Querying Approach for Reasoning About Situations Dheeraj Rajagopal Aman Madaan Niket Tandon Yiming Yang Shrimai Prabhumoye Abhilasha Ravichander Peter Clark Eduard H. Hovy ReLM LRM 16 6 0 01 Apr 2021
Self-Supervised Pretraining Improves Self-Supervised Pretraining Colorado Reed Xiangyu Yue Aniruddha Nrusimha Sayna Ebrahimi Vivek Vijaykumar ... Shanghang Zhang Devin Guillory Sean L. Metzger Kurt Keutzer Trevor Darrell 30 105 0 23 Mar 2021
Improving and Simplifying Pattern Exploiting Training Derek Tam Rakesh R Menon Joey Tianyi Zhou Shashank Srivastava Colin Raffel 21 149 0 22 Mar 2021
AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization Tiezheng Yu Zihan Liu Pascale Fung CLL 46 81 0 21 Mar 2021
CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review Dan Hendrycks Collin Burns Anya Chen Spencer Ball ELM AILaw 23 184 0 10 Mar 2021
Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Do P. Schramowski Cigdem Turan Nico Andersen Constantin Rothkopf Kristian Kersting 33 281 0 08 Mar 2021
"Sharks are not the threat humans are": Argument Component Segmentation in School Student Essays Tariq Alhindi Debanjan Ghosh 24 12 0 08 Mar 2021
Measuring Mathematical Problem Solving With the MATH Dataset Dan Hendrycks Collin Burns Saurav Kadavath Akul Arora Steven Basart Eric Tang D. Song Jacob Steinhardt ReLM FaML 84 1,891 0 05 Mar 2021
OAG-BERT: Towards A Unified Backbone Language Model For Academic Knowledge Services Xiao Liu Da Yin Jingnan Zheng Xingjian Zhang Peng Zhang Hongxia Yang Yuxiao Dong Jie Tang VLM 45 30 0 03 Mar 2021
Gradual Fine-Tuning for Low-Resource Domain Adaptation Haoran Xu Seth Ebner M. Yarmohammadi A. White Benjamin Van Durme Kenton W. Murray CLL 22 39 0 03 Mar 2021
ToxCCIn: Toxic Content Classification with Interpretability Tong Xiang Sean MacAvaney Eugene Yang Nazli Goharian 79 15 0 01 Mar 2021
ZJUKLAB at SemEval-2021 Task 4: Negative Augmentation with Language Model for Reading Comprehension of Abstract Meaning Xin Xie Xiangnan Chen Xiang Chen Yong Wang Ningyu Zhang Shumin Deng Huajun Chen 42 2 0 25 Feb 2021
BERT-based Acronym Disambiguation with Multiple Training Strategies Chunguang Pan Bingyan Song Shengguang Wang Zhipeng Luo 27 18 0 25 Feb 2021
LogME: Practical Assessment of Pre-trained Models for Transfer Learning Kaichao You Yong Liu Jianmin Wang Mingsheng Long 29 178 0 22 Feb 2021
Boosting Low-Resource Biomedical QA via Entity-Aware Masking Strategies Gabriele Pergola E. Kochkina Lin Gui Maria Liakata Yulan He 88 31 0 16 Feb 2021
Characterizing English Variation across Social Media Communities with BERT L. Lucy David Bamman 24 35 0 12 Feb 2021
Mind the Gap: Assessing Temporal Generalization in Neural Language Models Angeliki Lazaridou A. Kuncoro E. Gribovskaya Devang Agrawal Adam Liska ... Sebastian Ruder Dani Yogatama Kris Cao Susannah Young Phil Blunsom VLM 41 207 0 03 Feb 2021
AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuning Yuhan Liu Saurabh Agarwal Shivaram Venkataraman OffRL 19 54 0 02 Feb 2021
Word Alignment by Fine-tuning Embeddings on Parallel Corpora Zi-Yi Dou Graham Neubig 98 258 0 20 Jan 2021
Task Adaptive Pretraining of Transformers for Hostility Detection Tathagata Raha Sayar Ghosh Roy Ujwal Narayan Zubair Abid Vasudeva Varma 21 9 0 09 Jan 2021
Studying Strategically: Learning to Mask for Closed-book QA Qinyuan Ye Belinda Z. Li Sinong Wang Benjamin Bolte Hao Ma Wen-tau Yih Xiang Ren Madian Khabsa OffRL 27 11 0 31 Dec 2020
CoCoLM: COmplex COmmonsense Enhanced Language Model with Discourse Relations Changlong Yu Hongming Zhang Yangqiu Song Wilfred Ng 66 21 0 31 Dec 2020
Automated Lay Language Summarization of Biomedical Scientific Reviews Yue Guo Weijian Qiu Yizhong Wang T. Cohen 38 77 0 23 Dec 2020
MELINDA: A Multimodal Dataset for Biomedical Experiment Method Classification Te-Lin Wu Shikhar Singh S. Paul Gully A. Burns Nanyun Peng 30 18 0 16 Dec 2020
*-CFQ: Analyzing the Scalability of Machine Learning on a Compositional Task Dmitry Tsarkov Tibor Tihon Nathan Scales Nikola Momchev Danila Sinopalnikov Nathanael Scharli 18 17 0 15 Dec 2020
Causal BERT : Language models for causality detection between events expressed in text Vivek Khetan Roshni Ramnani M. Anand Shubhashis Sengupta Andrew E.Fano 22 44 0 10 Dec 2020
CrossNER: Evaluating Cross-Domain Named Entity Recognition Zihan Liu Yan Xu Tiezheng Yu Wenliang Dai Ziwei Ji Samuel Cahyawijaya Andrea Madotto Pascale Fung 78 146 0 08 Dec 2020
End-to-End QA on COVID-19: Domain Adaptation with Synthetic Training R. Reddy Bhavani Iyer Md Arafat Sultan Rong Zhang Avirup Sil Vittorio Castelli Radu Florian Salim Roukos OOD 20 19 0 02 Dec 2020
EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering Momchil Hardalov Todor Mihaylov Dimitrina Zlatkova Yoan Dinkov Ivan Koychev Preslav Nakov AI4Ed ELM 41 50 0 05 Nov 2020
CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Search Chenyan Xiong Zhenghao Liu Si Sun Zhuyun Dai Kaitao Zhang S. Yu Zhiyuan Liu Hoifung Poon Jianfeng Gao Paul N. Bennett 30 10 0 03 Nov 2020
WNUT-2020 Task 1 Overview: Extracting Entities and Relations from Wet Lab Protocols Jeniya Tabassum Sydney Lee Wei Xu Alan Ritter 18 18 0 27 Oct 2020
Unmasking Contextual Stereotypes: Measuring and Mitigating BERT's Gender Bias Marion Bartl Malvina Nissim Albert Gatt 28 123 0 27 Oct 2020
Rethinking embedding coupling in pre-trained language models Hyung Won Chung Thibault Févry Henry Tsai Melvin Johnson Sebastian Ruder 95 142 0 24 Oct 2020
Char2Subword: Extending the Subword Embedding Space Using Robust Character Compositionality Gustavo Aguilar Bryan McCann Tong Niu Nazneen Rajani N. Keskar Thamar Solorio 49 12 0 24 Oct 2020
HateBERT: Retraining BERT for Abusive Language Detection in English Tommaso Caselli Valerio Basile Jelena Mitrović Michael Granitzer 24 359 0 23 Oct 2020
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios Michael A. Hedderich Lukas Lange Heike Adel Jannik Strötgen Dietrich Klakow 221 287 0 23 Oct 2020
An Analysis of Simple Data Augmentation for Named Entity Recognition Xiang Dai Heike Adel 35 194 0 22 Oct 2020
Technical Question Answering across Tasks and Domains Wenhao Yu Lingfei Wu Yu Deng Qingkai Zeng R. Mahindru S. Guven Meng Jiang 36 8 0 19 Oct 2020
Pretrained Transformers for Text Ranking: BERT and Beyond Jimmy J. Lin Rodrigo Nogueira Andrew Yates VLM 244 612 0 13 Oct 2020
Zero-Shot Translation Quality Estimation with Explicit Cross-Lingual Patterns Lei Zhou Liang Ding Koichi Takeda 23 13 0 10 Oct 2020
Extracting a Knowledge Base of Mechanisms from COVID-19 Papers Tom Hope Aida Amini David Wadden Madeleine van Zuylen Sravanthi Parasa Eric Horvitz Daniel S. Weld Roy Schwartz Hannaneh Hajishirzi 34 29 0 08 Oct 2020
Resource-Enhanced Neural Model for Event Argument Extraction Jie Ma Shuai Wang Rishita Anubhai Miguel Ballesteros Yaser Al-Onaizan 16 39 0 06 Oct 2020
Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models Thuy-Trang Vu Dinh Q. Phung Gholamreza Haffari 22 24 0 05 Oct 2020
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media Xiang Dai Sarvnaz Karimi Ben Hachey Cécile Paris 24 35 0 02 Oct 2020
Data-Efficient Pretraining via Contrastive Self-Supervision Nils Rethmeier Isabelle Augenstein 28 20 0 02 Oct 2020
How to Motivate Your Dragon: Teaching Goal-Driven Agents to Speak and Act in Fantasy Worlds Prithviraj Ammanabrolu Jack Urbanek Margaret Li Arthur Szlam Tim Rocktaschel Jason Weston LM&Ro 19 44 0 01 Oct 2020
Understanding tables with intermediate pre-training Julian Martin Eisenschlos Syrine Krichene Thomas Müller LMTD 15 119 0 01 Oct 2020
Parsing with Multilingual BERT, a Small Corpus, and a Small Treebank Ethan C. Chau Lucy H. Lin Noah A. Smith 22 15 0 29 Sep 2020