AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing

12 August 2021

Katikapalli Subramanyam Kalyan

Papers citing "AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing"

50 / 231 papers shown

Title
Spanish Pre-trained BERT Model and Evaluation Data J. Cañete Gabriel Chaperon Rodrigo Fuentes Jou-Hui Ho Hojin Kang Jorge Pérez 71 662 0 06 Aug 2023
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing Pengfei Liu Weizhe Yuan Jinlan Fu Zhengbao Jiang Hiroaki Hayashi Graham Neubig VLM SyDa 195 3,964 0 28 Jul 2021
CausalBERT: Injecting Causal Knowledge Into Pre-trained Models with Minimal Supervision Zhongyang Li Xiao Ding Kuo Liao Bing Qin Ting Liu CML 95 19 0 21 Jul 2021
FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark Liang Xu Xiaojing Lu Chenyang Yuan Xuanwei Zhang Huilin Xu ... Guoao Wei X. Pan Xin Tian Libo Qin Hai Hu ELM 53 57 0 15 Jul 2021
FLEX: Unifying Evaluation for Few-Shot NLP Jonathan Bragg Arman Cohan Kyle Lo Iz Beltagy 253 106 0 15 Jul 2021
Deduplicating Training Data Makes Language Models Better Katherine Lee Daphne Ippolito A. Nystrom Chiyuan Zhang Douglas Eck Chris Callison-Burch Nicholas Carlini SyDa 354 628 0 14 Jul 2021
CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understanding Dong Wang Ning Ding Pijian Li Haitao Zheng AAML 55 117 0 01 Jul 2021
XLM-E: Cross-lingual Language Model Pre-training via ELECTRA Zewen Chi Shaohan Huang Li Dong Shuming Ma Bo Zheng ... Payal Bajaj Xia Song Xian-Ling Mao Heyan Huang Furu Wei 80 120 0 30 Jun 2021
A Closer Look at How Fine-tuning Changes BERT Yichu Zhou Vivek Srikumar 39 68 0 27 Jun 2021
Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains Yunzhi Yao Shaohan Huang Wenhui Wang Li Dong Furu Wei VLM ALM 58 48 0 25 Jun 2021
Towards Understanding and Mitigating Social Biases in Language Models Paul Pu Liang Chiyu Wu Louis-Philippe Morency Ruslan Salakhutdinov 93 388 0 24 Jun 2021
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization Yi Tay Vinh Q. Tran Sebastian Ruder Jai Gupta Hyung Won Chung Dara Bahri Zhen Qin Simon Baumgartner Cong Yu Donald Metzler 105 159 0 23 Jun 2021
BARTScore: Evaluating Generated Text as Text Generation Weizhe Yuan Graham Neubig Pengfei Liu 95 842 0 22 Jun 2021
CPM-2: Large-scale Cost-effective Pre-trained Language Models Zhengyan Zhang Yuxian Gu Xu Han Shengqi Chen Chaojun Xiao ... Minlie Huang Wentao Han Yang Liu Xiaoyan Zhu Maosong Sun MoE 66 87 0 20 Jun 2021
Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better Gaurav Menghani VLM MedIm 48 375 0 16 Jun 2021
Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning Bill Yuchen Lin Seyeon Lee Xiaoyang Qiao Xiang Ren ReLM LRM 54 62 0 13 Jun 2021
FastSeq: Make Sequence Generation Faster Yu Yan Fei Hu Jiusheng Chen Nikhil Bhendawade Ting Ye Yeyun Gong Nan Duan Desheng Cui Bingyu Chi Ruifei Zhang VLM 46 15 0 08 Jun 2021
A Survey of Transformers Tianyang Lin Yuxin Wang Xiangyang Liu Xipeng Qiu ViT 134 1,124 0 08 Jun 2021
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings Tatiana Likhomanenko Qiantong Xu Gabriel Synnaeve R. Collobert A. Rogozhnikov OOD ViT 60 59 0 06 Jun 2021
Knowledge Inheritance for Pre-trained Language Models Yujia Qin Yankai Lin Jing Yi Jiajie Zhang Xu Han ... Yusheng Su Zhiyuan Liu Peng Li Maosong Sun Jie Zhou VLM 57 50 0 28 May 2021
ByT5: Towards a token-free future with pre-trained byte-to-byte models Linting Xue Aditya Barua Noah Constant Rami Al-Rfou Sharan Narang Mihir Kale Adam Roberts Colin Raffel 87 503 0 28 May 2021
ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer Yuanmeng Yan Rumei Li Sirui Wang Fuzheng Zhang Wei Wu Weiran Xu SSL 108 556 0 25 May 2021
RobeCzech: Czech RoBERTa, a monolingual contextualized language representation model Milan Straka Jakub Náplava Jana Straková David Samuel 63 46 0 24 May 2021
KLUE: Korean Language Understanding Evaluation Sungjoon Park Jihyung Moon Sungdong Kim Won Ik Cho Jiyoon Han ... Seonghyun Kim Lucy Park Alice Oh Jung-Woo Ha Kyunghyun Cho ELM VLM 57 198 0 20 May 2021
CoTexT: Multi-task Learning with Code-Text Transformer Long Phan H. Tran Daniel Le Hieu Duy Nguyen J. Anibal Alec Peltekian Yanfang Ye 59 136 0 18 May 2021
MathBERT: A Pre-Trained Model for Mathematical Formula Understanding Shuai Peng Ke Yuan Liangcai Gao Zhi Tang AIMat 78 108 0 02 May 2021
PanGu- $α$ : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation Wei Zeng Xiaozhe Ren Teng Su Hui Wang Yi-Lun Liao ... Gaojun Fan Yaowei Wang Xuefeng Jin Qun Liu Yonghong Tian ALM MoE AI4CE 69 213 0 26 Apr 2021
Efficient pre-training objectives for Transformers Luca Di Liello Matteo Gabburo Alessandro Moschitti 18 15 0 20 Apr 2021
SimCSE: Simple Contrastive Learning of Sentence Embeddings Tianyu Gao Xingcheng Yao Danqi Chen AILaw SSL 261 3,386 0 18 Apr 2021
MT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs Zewen Chi Li Dong Shuming Ma Shaohan Huang Xian-Ling Mao Heyan Huang Furu Wei LRM 72 73 0 18 Apr 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 533 4,032 0 18 Apr 2021
Dual-View Distilled BERT for Sentence Embedding Xingyi Cheng 3DV 27 14 0 18 Apr 2021
What to Pre-Train on? Efficient Intermediate Task Selection Clifton A. Poth Jonas Pfeiffer Andreas Rucklé Iryna Gurevych 59 100 0 16 Apr 2021
Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders Fangyu Liu Ivan Vulić Anna Korhonen Nigel Collier VLM OffRL 73 120 0 16 Apr 2021
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation Sebastian Ruder Noah Constant Jan A. Botha Aditya Siddhant Orhan Firat ... Pengfei Liu Junjie Hu Dan Garrette Graham Neubig Melvin Johnson ELM AAML LRM 57 187 0 15 Apr 2021
TSDAE: Using Transformer-based Sequential Denoising Auto-Encoder for Unsupervised Sentence Embedding Learning Kexin Wang Nils Reimers Iryna Gurevych 130 184 0 14 Apr 2021
ExplainaBoard: An Explainable Leaderboard for NLP Pengfei Liu Jinlan Fu Yanghua Xiao Weizhe Yuan Shuaichen Chang Junqi Dai Yixin Liu Zihuiwen Ye Zi-Yi Dou Graham Neubig XAI LRM ELM 64 55 0 13 Apr 2021
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages Gowtham Ramesh Sumanth Doddapaneni Aravinth Bheemaraj Mayank Jobanputra AK Raghavan ... K. Deepak Vivek Raghavan Anoop Kunchukuttan Pratyush Kumar Mitesh Khapra LRM 70 235 0 12 Apr 2021
IndT5: A Text-to-Text Transformer for 10 Indigenous Languages El Moatez Billah Nagoudi Wei-Rui Chen Muhammad Abdul-Mageed H. Cavusoglu 55 24 0 04 Apr 2021
MuRIL: Multilingual Representations for Indian Languages Simran Khanuja Diksha Bansal Sarvesh Mehtani Savya Khosla Atreyee Dey ... Shachi Dave Shruti Gupta Subhash Chandra Bose Gali Vishnu Subramanian Partha P. Talukdar 73 289 0 19 Mar 2021
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation J. Clark Dan Garrette Iulia Turc John Wieting 89 220 0 11 Mar 2021
Unified Pre-training for Program Understanding and Generation Wasi Uddin Ahmad Saikat Chakraborty Baishakhi Ray Kai-Wei Chang 125 765 0 10 Mar 2021
Does He Wink or Does He Nod? A Challenging Benchmark for Evaluating Word Understanding of Language Models Lutfi Kerem Senel Hinrich Schütze 43 5 0 06 Feb 2021
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics Sebastian Gehrmann Tosin Adewumi Karmanya Aggarwal Pawan Sasanka Ammanamanchi Aremu Anuoluwapo ... Nishant Subramani Wei Xu Diyi Yang Akhila Yerukola Jiawei Zhou VLM 303 286 0 02 Feb 2021
Multilingual LAMA: Investigating Knowledge in Multilingual Pretrained Language Models Nora Kassner Philipp Dufter Hinrich Schütze 69 141 0 01 Feb 2021
KoreALBERT: Pretraining a Lite BERT Model for Korean Language Understanding HyunJae Lee Jaewoong Yoon Bonggyu Hwang Seongho Joe Seungjai Min Youngjune Gwon SSeg 45 16 0 27 Jan 2021
WangchanBERTa: Pretraining transformer-based Thai Language Models Lalita Lowphansirikul Charin Polpanumas Nawat Jantrakulchai Sarana Nutanong 30 76 0 24 Jan 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 85 2,178 0 11 Jan 2021
Transformers in Vision: A Survey Salman Khan Muzammal Naseer Munawar Hayat Syed Waqas Zamir Fahad Shahbaz Khan M. Shah ViT 294 2,503 0 04 Jan 2021
Prefix-Tuning: Optimizing Continuous Prompts for Generation Xiang Lisa Li Percy Liang 221 4,247 0 01 Jan 2021