Pre-trained Models for Natural Language Processing: A Survey

18 March 2020

Xipeng Qiu

Tianxiang Sun

Xuanjing Huang

Papers citing "Pre-trained Models for Natural Language Processing: A Survey"

50 / 91 papers shown

Title
Handling Symbolic Language in Student Texts: A Comparative Study of NLP Embedding Models Tom Bleckmann Paul Tschisgale 123 0 0 23 May 2025
Can Large Language Models Understand Intermediate Representations in Compilers? Hailong Jiang Jianfeng Zhu Yao Wan B. Fang Hongyu Zhang Ruoming Jin Qiang Guan 75 1 0 07 Feb 2025
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics Kai He Rui Mao Qika Lin Yucheng Ruan Xiang Lan Mengling Feng Min Zhang LM&MA AILaw 165 166 0 28 Jan 2025
Human Activity Recognition in an Open World D. Prijatelj Samuel Grieggs Jin Huang Dawei Du Ameya Shringi Christopher Funk Adam Kaufman Eric Robertson Walter J. Scheirer University of Notre Dame 109 3 0 17 Jan 2025
Mathematical Language Models: A Survey Wen Liu Hanglei Hu Jie Zhou Yuyang Ding Junsong Li ... Mengliang He Qin Chen Bo Jiang Aimin Zhou Liang He LRM 146 14 0 03 Jan 2025
PortLLM: Personalizing Evolving Large Language Models with Training-Free and Portable Model Patches Rana Muhammad Shahroz Khan Pingzhi Li Sukwon Yun Zhenyu Wang S. Nirjon Chau-Wai Wong Tianlong Chen KELM 83 2 0 08 Oct 2024
A Large-Scale Study of Model Integration in ML-Enabled Software Systems Yorick Sens Henriette Knopp Sven Peldszus Thorsten Berger AIFin 52 2 0 12 Aug 2024
Who Wrote This? The Key to Zero-Shot LLM-Generated Text Detection Is GECScore Junchao Wu Runzhe Zhan Derek F. Wong Shu Yang Xuebo Liu Lidia S. Chao Min Zhang DeLMO 88 4 0 07 May 2024
A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods Hanlei Jin Yang Zhang Dan Meng Jun Wang Jinghua Tan 157 87 0 05 Mar 2024
Large Language Models: A Survey Shervin Minaee Tomas Mikolov Narjes Nikzad M. Asgari-Chenaghlu R. Socher Xavier Amatriain Jianfeng Gao ALM LM&MA ELM 171 389 0 09 Feb 2024
Large Language Models as Topological Structure Enhancers for Text-Attributed Graphs Shengyin Sun Yuxiang Ren Chen Ma Xuecang Zhang 181 21 0 24 Nov 2023
Early Exiting with Ensemble Internal Classifiers Tianxiang Sun Yunhua Zhou Xiangyang Liu Xinyu Zhang Hao Jiang Bo Zhao Xuanjing Huang Xipeng Qiu 47 30 0 28 May 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 474 3,952 0 18 Apr 2021
Factual Probing Is [MASK]: Learning vs. Learning to Recall Zexuan Zhong Dan Friedman Danqi Chen 42 407 0 12 Apr 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 182 4,209 0 01 Jan 2021
WARP: Word-level Adversarial ReProgramming Karen Hambardzumyan Hrant Khachatrian Jonathan May AAML 291 345 0 01 Jan 2021
It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners Timo Schick Hinrich Schütze 92 970 0 15 Sep 2020
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices Zhiqing Sun Hongkun Yu Xiaodan Song Renjie Liu Yiming Yang Denny Zhou MQ 93 807 0 06 Apr 2020
FastBERT: a Self-distilling BERT with Adaptive Inference Time Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Haotang Deng Qi Ju 78 356 0 05 Apr 2020
A Survey on Contextual Embeddings Qi Liu Matt J. Kusner Phil Blunsom 234 146 0 16 Mar 2020
AraBERT: Transformer-based Model for Arabic Language Understanding Wissam Antoun Fady Baly Hazem M. Hajj 91 959 0 28 Feb 2020
TextBrewer: An Open-Source Knowledge Distillation Toolkit for Natural Language Processing Ziqing Yang Yiming Cui Zhipeng Chen Wanxiang Che Ting Liu Shijin Wang Guoping Hu VLM 54 47 0 28 Feb 2020
Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT Lichao Sun Kazuma Hashimoto Wenpeng Yin Akari Asai Jia Li Philip Yu Caiming Xiong SILM AAML 44 101 0 27 Feb 2020
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping Jesse Dodge Gabriel Ilharco Roy Schwartz Ali Farhadi Hannaneh Hajishirzi Noah A. Smith 81 594 0 15 Feb 2020
Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference Youwei Song Jiahai Wang Zhiwei Liang Zhiyue Liu Tao Jiang 49 76 0 12 Feb 2020
Adversarial Training for Aspect-Based Sentiment Analysis with BERT Akbar Karimi L. Rossi Andrea Prati 224 99 0 30 Jan 2020
Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Induction Taeuk Kim Jihun Choi Daniel Edmiston Sang-goo Lee 52 90 0 30 Jan 2020
Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference Timo Schick Hinrich Schütze 310 1,606 0 21 Jan 2020
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization Jingqing Zhang Yao-Min Zhao Mohammad Saleh Peter J. Liu RALM 3DGS 214 2,029 0 18 Dec 2019
Multilingual is not enough: BERT for Finnish Antti Virtanen Jenna Kanerva Rami Ilo Jouni Luoma Juhani Luotolahti T. Salakoski Filip Ginter S. Pyysalo 74 280 0 15 Dec 2019
Inducing Relational Knowledge from BERT Zied Bouraoui Jose Camacho-Collados Steven Schockaert 73 167 0 28 Nov 2019
How Can We Know What Language Models Know? Zhengbao Jiang Frank F. Xu Jun Araki Graham Neubig KELM 101 1,396 0 28 Nov 2019
TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection Siddhant Garg Thuy Vu Alessandro Moschitti 60 215 0 11 Nov 2019
CamemBERT: a Tasty French Language Model Louis Martin Benjamin Muller Pedro Ortiz Suarez Yoann Dupont Laurent Romary Eric Villemonte de la Clergerie Djamé Seddah Benoît Sagot 92 969 0 10 Nov 2019
Unsupervised Cross-lingual Representation Learning at Scale Alexis Conneau Kartikay Khandelwal Naman Goyal Vishrav Chaudhary Guillaume Wenzek Francisco Guzmán Edouard Grave Myle Ott Luke Zettlemoyer Veselin Stoyanov 166 6,496 0 05 Nov 2019
Select, Answer and Explain: Interpretable Multi-hop Reading Comprehension over Multiple Documents Ming Tu Kevin Huang Guangtao Wang Jing-ling Huang Xiaodong He Bowen Zhou RALM 59 144 0 01 Nov 2019
A Mutual Information Maximization Perspective of Language Representation Learning Lingpeng Kong Cyprien de Masson dÁutume Wang Ling Lei Yu Zihang Dai Dani Yogatama SSL 248 167 0 18 Oct 2019
Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT Matthew Tang P. Gandhi Md. Ahsanul Kabir C. Zou J. Blakey Xiao Luo 37 39 0 13 Oct 2019
DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter Victor Sanh Lysandre Debut Julien Chaumond Thomas Wolf 143 7,437 0 02 Oct 2019
Exploiting BERT for End-to-End Aspect-based Sentiment Analysis Xin Li Lidong Bing Wenxuan Zhang W. Lam 61 278 0 02 Oct 2019
Specializing Word Embeddings (for Parsing) by Information Bottleneck Xiang Lisa Li Jason Eisner 52 66 0 01 Oct 2019
On the use of BERT for Neural Machine Translation Stéphane Clinchant K. Jung Vassilina Nikoulina 40 90 0 27 Sep 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 279 6,420 0 26 Sep 2019
FreeLB: Enhanced Adversarial Training for Natural Language Understanding Chen Zhu Yu Cheng Zhe Gan S. Sun Tom Goldstein Jingjing Liu AAML 251 440 0 25 Sep 2019
UNITER: UNiversal Image-TExt Representation Learning Yen-Chun Chen Linjie Li Licheng Yu Ahmed El Kholy Faisal Ahmed Zhe Gan Yu Cheng Jingjing Liu VLM OT 87 447 0 25 Sep 2019
Cross-Lingual Natural Language Generation via Pre-Training Zewen Chi Li Dong Furu Wei Wenhui Wang Xian-Ling Mao Heyan Huang 51 136 0 23 Sep 2019
TinyBERT: Distilling BERT for Natural Language Understanding Xiaoqi Jiao Yichun Yin Lifeng Shang Xin Jiang Xiao Chen Linlin Li F. Wang Qun Liu VLM 65 1,847 0 23 Sep 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Mohammad Shoeybi M. Patwary Raul Puri P. LeGresley Jared Casper Bryan Catanzaro MoE 293 1,861 0 17 Sep 2019
K-BERT: Enabling Language Representation with Knowledge Graph Weijie Liu Peng Zhou Zhe Zhao Zhiruo Wang Qi Ju Haotang Deng Ping Wang 274 785 0 17 Sep 2019
Language Models as Knowledge Bases? Fabio Petroni Tim Rocktaschel Patrick Lewis A. Bakhtin Yuxiang Wu Alexander H. Miller Sebastian Riedel KELM AI4MH 535 2,639 0 03 Sep 2019