Pre-Training a Language Model Without Human Language

22 December 2020

Papers citing "Pre-Training a Language Model Without Human Language"

12 / 12 papers shown

Title
When Do You Need Billions of Words of Pretraining Data? Yian Zhang Alex Warstadt Haau-Sing Li Samuel R. Bowman 41 139 0 10 Nov 2020
On the importance of pre-training data volume for compact language models Vincent Micheli Martin d'Hoffschmidt Franccois Fleuret 25 42 0 08 Oct 2020
A Monolingual Approach to Contextualized Word Embeddings for Mid-Resource Languages Pedro Ortiz Suarez Laurent Romary Benoît Sagot 47 228 0 11 Jun 2020
Don't Stop Pretraining: Adapt Language Models to Domains and Tasks Suchin Gururangan Ana Marasović Swabha Swayamdipta Kyle Lo Iz Beltagy Doug Downey Noah A. Smith VLM AI4CE CLL 98 2,380 0 23 Apr 2020
Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Information Seonwoo Min Seunghyun Park Siwon Kim Hyun-Soo Choi Byunghan Lee Sungroh Yoon SSL 34 62 0 25 Nov 2019
Revealing the Dark Secrets of BERT Olga Kovaleva Alexey Romanov Anna Rogers Anna Rumshisky 24 551 0 21 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 364 24,160 0 26 Jul 2019
Open Sesame: Getting Inside BERT's Linguistic Knowledge Yongjie Lin Y. Tan Robert Frank 23 287 0 04 Jun 2019
What do you learn from context? Probing for sentence structure in contextualized word representations Ian Tenney Patrick Xia Berlin Chen Alex Jinpeng Wang Adam Poliak ... Najoung Kim Benjamin Van Durme Samuel R. Bowman Dipanjan Das Ellie Pavlick 154 852 0 15 May 2019
BERT Rediscovers the Classical NLP Pipeline Ian Tenney Dipanjan Das Ellie Pavlick MILM SSeg 83 1,458 0 15 May 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 821 93,936 0 11 Oct 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 570 7,080 0 20 Apr 2018