Pre-trained Models for Natural Language Processing: A Survey

18 March 2020

Xipeng Qiu

Tianxiang Sun

Xuanjing Huang

Papers citing "Pre-trained Models for Natural Language Processing: A Survey"

41 / 91 papers shown

Title
Commonsense Knowledge Mining from Pretrained Models Joshua Feldman Joe Davison Alexander M. Rush SSL 72 330 0 02 Sep 2019
Small and Practical BERT Models for Sequence Labeling Henry Tsai Jason Riesa Melvin Johnson N. Arivazhagan Xin Li Amelia Archer VLM 41 121 0 31 Aug 2019
VL-BERT: Pre-training of Generic Visual-Linguistic Representations Weijie Su Xizhou Zhu Yue Cao Bin Li Lewei Lu Furu Wei Jifeng Dai VLM MLLM SSL 126 1,657 0 22 Aug 2019
Multi-Task Self-Supervised Learning for Disfluency Detection Shaolei Wang Wanxiang Che Qi Liu Pengda Qin Ting Liu William Yang Wang SSL 34 56 0 15 Aug 2019
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding Wei Wang Bin Bi Ming Yan Chen Henry Wu Zuyi Bao Jiangnan Xia Liwei Peng Luo Si 52 260 0 13 Aug 2019
VisualBERT: A Simple and Performant Baseline for Vision and Language Liunian Harold Li Mark Yatskar Da Yin Cho-Jui Hsieh Kai-Wei Chang VLM 111 1,939 0 09 Aug 2019
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks Jiasen Lu Dhruv Batra Devi Parikh Stefan Lee SSL VLM 201 3,659 0 06 Aug 2019
Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment Di Jin Zhijing Jin Qiufeng Wang Peter Szolovits SILM AAML 127 1,064 0 27 Jul 2019
SpanBERT: Improving Pre-training by Representing and Predicting Spans Mandar Joshi Danqi Chen Yinhan Liu Daniel S. Weld Luke Zettlemoyer Omer Levy 122 1,953 0 24 Jul 2019
Are Sixteen Heads Really Better than One? Paul Michel Omer Levy Graham Neubig MoE 90 1,051 0 25 May 2019
Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned Elena Voita David Talbot F. Moiseev Rico Sennrich Ivan Titov 76 1,120 0 23 May 2019
Story Ending Prediction by Transferable BERT Zhongyang Li Xiao Ding Ting Liu 58 52 0 17 May 2019
Adaptation of Deep Bidirectional Multilingual Transformers for Russian Language Yuri Kuratov M. Arkhipov 58 275 0 17 May 2019
ERNIE: Enhanced Language Representation with Informative Entities Zhengyan Zhang Xu Han Zhiyuan Liu Xin Jiang Maosong Sun Qun Liu 77 1,390 0 17 May 2019
What do you learn from context? Probing for sentence structure in contextualized word representations Ian Tenney Patrick Xia Berlin Chen Alex Jinpeng Wang Adam Poliak ... Najoung Kim Benjamin Van Durme Samuel R. Bowman Dipanjan Das Ellie Pavlick 161 853 0 15 May 2019
BERT Rediscovers the Classical NLP Pipeline Ian Tenney Dipanjan Das Ellie Pavlick MILM SSeg 107 1,458 0 15 May 2019
How to Fine-Tune BERT for Text Classification? Chi Sun Xipeng Qiu Yige Xu Xuanjing Huang 69 1,508 0 14 May 2019
Unified Language Model Pre-training for Natural Language Understanding and Generation Li Dong Nan Yang Wenhui Wang Furu Wei Xiaodong Liu Yu Wang Jianfeng Gao M. Zhou H. Hon ELM AI4CE 163 1,553 0 08 May 2019
MASS: Masked Sequence to Sequence Pre-training for Language Generation Kaitao Song Xu Tan Tao Qin Jianfeng Lu Tie-Yan Liu 94 962 0 07 May 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 192 2,296 0 02 May 2019
ERNIE: Enhanced Representation through Knowledge Integration Yu Sun Shuohuan Wang Yukun Li Shikun Feng Xuyi Chen Han Zhang Xin Tian Danxiang Zhu Hao Tian Hua Wu 102 898 0 19 Apr 2019
Publicly Available Clinical BERT Embeddings Emily Alsentzer John R. Murphy Willie Boag W. Weng Di Jin Tristan Naumann Matthew B. A. McDermott AI4MH 125 1,956 0 06 Apr 2019
SciBERT: A Pretrained Language Model for Scientific Text Iz Beltagy Kyle Lo Arman Cohan 88 2,948 0 26 Mar 2019
Utilizing BERT for Aspect-Based Sentiment Analysis via Constructing Auxiliary Sentence Chi Sun Luyao Huang Xipeng Qiu 52 607 0 22 Mar 2019
To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Tasks Matthew E. Peters Sebastian Ruder Noah A. Smith 65 435 0 14 Mar 2019
Attention is not Explanation Sarthak Jain Byron C. Wallace FAtt 87 1,307 0 26 Feb 2019
Star-Transformer Qipeng Guo Xipeng Qiu Pengfei Liu Yunfan Shao Xiangyang Xue Zheng Zhang 58 264 0 25 Feb 2019
BioBERT: a pre-trained biomedical language representation model for biomedical text mining Jinhyuk Lee Wonjin Yoon Sungdong Kim Donghyeon Kim Sunkyu Kim Chan Ho So Jaewoo Kang OOD 118 5,579 0 25 Jan 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 146 3,714 0 09 Jan 2019
CoQA: A Conversational Question Answering Challenge Siva Reddy Danqi Chen Christopher D. Manning RALM HAI 86 1,197 0 21 Aug 2018
Exploiting Semantics in Neural Machine Translation with Graph Convolutional Networks Diego Marcheggiani Jasmijn Bastings Ivan Titov GNN NAI 80 186 0 23 Apr 2018
Learned in Translation: Contextualized Word Vectors Bryan McCann James Bradbury Caiming Xiong R. Socher 107 907 0 01 Aug 2017
Knowledge Graph Representation with Jointly Structural and Textual Encoding Jiacheng Xu Kan Chen Xipeng Qiu Xuanjing Huang 3DV 51 126 0 26 Nov 2016
Unsupervised Pretraining for Sequence to Sequence Learning Prajit Ramachandran Peter J. Liu Quoc V. Le SSL AIMat 71 281 0 08 Nov 2016
Enriching Word Vectors with Subword Information Piotr Bojanowski Edouard Grave Armand Joulin Tomas Mikolov NAI SSL VLM 198 9,944 0 15 Jul 2016
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 173 8,067 0 16 Jun 2016
Adaptive Computation Time for Recurrent Neural Networks Alex Graves 54 544 0 29 Mar 2016
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 157 7,683 0 31 Aug 2015
Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling Junyoung Chung Çağlar Gülçehre Kyunghyun Cho Yoshua Bengio 328 12,662 0 11 Dec 2014
A Convolutional Neural Network for Modelling Sentences Nal Kalchbrenner Edward Grefenstette Phil Blunsom 82 3,556 0 08 Apr 2014
Representation Learning: A Review and New Perspectives Yoshua Bengio Aaron Courville Pascal Vincent OOD SSL 190 12,384 0 24 Jun 2012