RoBERTa: A Robustly Optimized BERT Pretraining Approach

26 July 2019

Luke Zettlemoyer

Papers citing "RoBERTa: A Robustly Optimized BERT Pretraining Approach"

50 / 4,659 papers shown

Title
Adversarial Filters of Dataset Biases Ronan Le Bras Swabha Swayamdipta Chandra Bhagavatula Rowan Zellers Matthew E. Peters Ashish Sabharwal Yejin Choi 36 220 0 10 Feb 2020
REALM: Retrieval-Augmented Language Model Pre-Training Kelvin Guu Kenton Lee Zora Tung Panupong Pasupat Ming-Wei Chang RALM 48 2,006 0 10 Feb 2020
Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts Max Ryabinin Anton I. Gusev FedML 27 48 0 10 Feb 2020
Pre-training Tasks for Embedding-based Large-scale Retrieval Wei-Cheng Chang Felix X. Yu Yin-Wen Chang Yiming Yang Sanjiv Kumar RALM 13 301 0 10 Feb 2020
Segmented Graph-Bert for Graph Instance Modeling Jiawei Zhang SSeg 25 6 0 09 Feb 2020
MA-DST: Multi-Attention Based Scalable Dialog State Tracking Adarsh Kumar Peter Ku Anuj Kumar Goyal A. Metallinou Dilek Z. Hakkani-Tür 27 58 0 07 Feb 2020
perm2vec: Graph Permutation Selection for Decoding of Error Correction Codes using Self-Attention Nir Raviv Avi Caciularu Tomer Raviv Jacob Goldberger Yair Be’ery 23 8 0 06 Feb 2020
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters Ruize Wang Duyu Tang Nan Duan Zhongyu Wei Xuanjing Huang Jianshu Ji Guihong Cao Daxin Jiang Ming Zhou KELM 48 545 0 05 Feb 2020
Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension Max Bartolo A. Roberts Johannes Welbl Sebastian Riedel Pontus Stenetorp AAML 37 167 0 02 Feb 2020
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction Taeuk Kim Jihun Choi Daniel Edmiston Sang-goo Lee 22 90 0 30 Jan 2020
Retrospective Reader for Machine Reading Comprehension ZhuoSheng Zhang Junjie Yang Hai Zhao RALM 25 226 0 27 Jan 2020
DUMA: Reading Comprehension with Transposition Thinking Pengfei Zhu Hai Zhao Xiaoguang Li AI4CE 34 35 0 26 Jan 2020
Generating Representative Headlines for News Stories Xiaotao Gu Yuning Mao Jiawei Han Jialu Liu Hongkun Yu You Wu Cong Yu Daniel Finnie Jiaqi Zhai Nicholas Zukoski 30 70 0 26 Jan 2020
TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval Jie Lei Licheng Yu Tamara L. Berg Joey Tianyi Zhou 119 276 0 24 Jan 2020
Multilingual Denoising Pre-training for Neural Machine Translation Yinhan Liu Jiatao Gu Naman Goyal Xian Li Sergey Edunov Marjan Ghazvininejad M. Lewis Luke Zettlemoyer AI4CE AIMat 55 1,773 0 22 Jan 2020
ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Data Di Qi Lin Su Jianwei Song Edward Cui Taroon Bharti Arun Sacheti VLM 40 259 0 22 Jan 2020
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Timo Schick Hinrich Schütze 258 1,591 0 21 Jan 2020
RobBERT: a Dutch RoBERTa-based Language Model Pieter Delobelle Thomas Winters Bettina Berendt 18 233 0 17 Jan 2020
CLUENER2020: Fine-grained Named Entity Recognition Dataset and Benchmark for Chinese Liang Xu Yu Tong Qianqian Dong Yixuan Liao Cong Yu Yin Tian Weitang Liu Lu Li Caiquan Liu Xuanwei Zhang 32 48 0 13 Jan 2020
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training Weizhen Qi Yu Yan Yeyun Gong Dayiheng Liu Nan Duan Jiusheng Chen Ruofei Zhang Ming Zhou AI4TS 27 446 0 13 Jan 2020
LayoutLM: Pre-training of Text and Layout for Document Image Understanding Yiheng Xu Minghao Li Lei Cui Shaohan Huang Furu Wei Ming Zhou 26 686 0 31 Dec 2019
oLMpics -- On what Language Model Pre-training Captures Alon Talmor Yanai Elazar Yoav Goldberg Jonathan Berant LRM 34 300 0 31 Dec 2019
Are Transformers universal approximators of sequence-to-sequence functions? Chulhee Yun Srinadh Bhojanapalli A. S. Rawat Sashank J. Reddi Sanjiv Kumar 26 336 0 20 Dec 2019
Pretrained Encyclopedia: Weakly Supervised Knowledge-Pretrained Language Model Wenhan Xiong Jingfei Du William Yang Wang Veselin Stoyanov SSL KELM 52 201 0 20 Dec 2019
BERTje: A Dutch BERT Model Wietse de Vries Andreas van Cranenburgh Arianna Bisazza Tommaso Caselli Gertjan van Noord Malvina Nissim VLM SSeg 25 291 0 19 Dec 2019
Cross-Lingual Ability of Multilingual BERT: An Empirical Study Karthikeyan K Zihan Wang Stephen D. Mayhew Dan Roth LRM 36 333 0 17 Dec 2019
Multilingual is not enough: BERT for Finnish Antti Virtanen Jenna Kanerva Rami Ilo Jouni Luoma Juhani Luotolahti T. Salakoski Filip Ginter S. Pyysalo 36 277 0 15 Dec 2019
FlauBERT: Unsupervised Language Model Pre-training for French Hang Le Loïc Vial Jibril Frej Vincent Segonne Maximin Coavoux Benjamin Lecouteux A. Allauzen Benoît Crabbé Laurent Besacier D. Schwab AI4CE 49 395 0 11 Dec 2019
Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline Vishvak Murahari Dhruv Batra Devi Parikh Abhishek Das VLM 23 115 0 05 Dec 2019
A Transformer-based approach to Irony and Sarcasm detection Rolandos Alexandros Potamias Georgios Siolas A. Stafylopatis 33 206 0 23 Nov 2019
Global Greedy Dependency Parsing Z. Li Zhao Hai Kevin Parnow 31 31 0 20 Nov 2019
The Eighth Dialog System Technology Challenge Seokhwan Kim Michel Galley Chulaka Gunasekara Sungjin Lee Adam Atkinson ... Tim K. Marks Abhinav Rastogi Xiaoxue Zang Srinivas Sunkara Raghav Gupta VLM 18 65 0 14 Nov 2019
Sato: Contextual Semantic Type Detection in Tables Dan Zhang Yoshihiko Suhara Jinfeng Li Madelon Hulsebos cCaugatay Demiralp W. Tan LMTD 16 15 0 14 Nov 2019
What do you mean, BERT? Assessing BERT as a Distributional Semantics Model Timothee Mickus Denis Paperno Mathieu Constant Kees van Deemter 26 45 0 13 Nov 2019
Adapting and evaluating a deep learning language model for clinical why-question answering Andrew Wen Mohamed Y. Elwazir Sungrim Moon Jungwei Fan LM&MA 24 31 0 13 Nov 2019
Neural Duplicate Question Detection without Labeled Training Data Andreas Rucklé N. Moosavi Iryna Gurevych OOD AAML 19 11 0 13 Nov 2019
Attending to Entities for Better Text Understanding Pengxiang Cheng K. Erk LRM 24 37 0 11 Nov 2019
Improving BERT Fine-tuning with Embedding Normalization Wenxuan Zhou Junyi Du Xiang Ren 18 6 0 10 Nov 2019
Effectiveness of self-supervised pre-training for speech recognition Alexei Baevski Michael Auli Abdel-rahman Mohamed SSL 27 147 0 10 Nov 2019
CamemBERT: a Tasty French Language Model Louis Martin Benjamin Muller Pedro Ortiz Suarez Yoann Dupont Laurent Romary Eric Villemonte de la Clergerie Djamé Seddah Benoît Sagot 42 956 0 10 Nov 2019
ConveRT: Efficient and Accurate Conversational Representations from Transformers Matthew Henderson I. Casanueva Nikola Mrkvsić Pei-hao Su Tsung-Hsien Ivan Vulić 21 196 0 09 Nov 2019
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT Nina Poerner Ulli Waltinger Hinrich Schütze 15 156 0 09 Nov 2019
Improving Machine Reading Comprehension via Adversarial Training Ziqing Yang Yiming Cui Wanxiang Che Ting Liu Shijin Wang Guoping Hu 27 17 0 09 Nov 2019
How Decoding Strategies Affect the Verifiability of Generated Text Luca Massarelli Fabio Petroni Aleksandra Piktus Myle Ott Tim Rocktaschel Vassilis Plachouras Fabrizio Silvestri Sebastian Riedel 23 50 0 09 Nov 2019
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly Nora Kassner Hinrich Schütze 26 316 0 08 Nov 2019
What Would Elsa Do? Freezing Layers During Transformer Fine-Tuning Jaejun Lee Raphael Tang Jimmy J. Lin 34 121 0 08 Nov 2019
Certified Data Removal from Machine Learning Models Chuan Guo Tom Goldstein Awni Y. Hannun Laurens van der Maaten MU 52 420 0 08 Nov 2019
The TechQA Dataset Vittorio Castelli Rishav Chakravarti Saswati Dana Anthony Ferritto Radu Florian ... Andrzej Sakrajda Avirup Sil Rosario A. Uceda-Sosa T. Ward Rong Zhang 24 45 0 08 Nov 2019
Blockwise Self-Attention for Long Document Understanding J. Qiu Hao Ma Omer Levy Scott Yih Sinong Wang Jie Tang 11 252 0 07 Nov 2019
S2ORC: The Semantic Scholar Open Research Corpus Kyle Lo Lucy Lu Wang Mark Neumann Rodney Michael Kinney Daniel S. Weld OffRL AI4CE 39 10 0 07 Nov 2019