On the validity of pre-trained transformers for natural language processing in the software engineering domain

10 September 2021

Papers citing "On the validity of pre-trained transformers for natural language processing in the software engineering domain"

30 / 30 papers shown

Title
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 83 2,168 0 11 Jan 2021
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 499 2,074 0 28 Jul 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 662 41,736 0 28 May 2020
Code and Named Entity Recognition in StackOverflow Jeniya Tabassum Mounica Maddela Wei Xu Alan Ritter 103 116 0 04 May 2020
CodeBERT: A Pre-Trained Model for Programming and Natural Languages Zhangyin Feng Daya Guo Duyu Tang Nan Duan Xiaocheng Feng ... Linjun Shou Bing Qin Ting Liu Daxin Jiang Ming Zhou 143 2,613 0 19 Feb 2020
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 213 10,792 0 29 Oct 2019
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations Zhenzhong Lan Mingda Chen Sebastian Goodman Kevin Gimpel Piyush Sharma Radu Soricut SSL AIMat 326 6,441 0 26 Sep 2019
FinBERT: Financial Sentiment Analysis with Pre-trained Language Models Dogu Araci AIFin 105 641 0 27 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 520 24,351 0 26 Jul 2019
XLNet: Generalized Autoregressive Pretraining for Language Understanding Zhilin Yang Zihang Dai Yiming Yang J. Carbonell Ruslan Salakhutdinov Quoc V. Le AI4CE 218 8,415 0 19 Jun 2019
A Multiscale Visualization of Attention in the Transformer Model Jesse Vig ViT 77 577 0 12 Jun 2019
Energy and Policy Considerations for Deep Learning in NLP Emma Strubell Ananya Ganesh Andrew McCallum 62 2,647 0 05 Jun 2019
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems Alex Jinpeng Wang Yada Pruksachatkun Nikita Nangia Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 234 2,307 0 02 May 2019
ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission Kexin Huang Jaan Altosaar Rajesh Ranganath OOD 91 899 0 10 Apr 2019
Large Batch Optimization for Deep Learning: Training BERT in 76 minutes Yang You Jing Li Sashank J. Reddi Jonathan Hseu Sanjiv Kumar Srinadh Bhojanapalli Xiaodan Song J. Demmel Kurt Keutzer Cho-Jui Hsieh ODL 210 993 0 01 Apr 2019
Import2vec - Learning Embeddings for Software Libraries B. Theeten Frederik Vandeputte Tom Van Cutsem SSL 36 34 0 27 Mar 2019
SciBERT: A Pretrained Language Model for Scientific Text Iz Beltagy Kyle Lo Arman Cohan 118 2,957 0 26 Mar 2019
BioBERT: a pre-trained biomedical language representation model for biomedical text mining Jinhyuk Lee Wonjin Yoon Sungdong Kim Donghyeon Kim Sunkyu Kim Chan Ho So Jaewoo Kang OOD 140 5,628 0 25 Jan 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 1.5K 94,511 0 11 Oct 2018
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems S. Kiritchenko Saif M. Mohammad FaML 77 436 0 11 May 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 904 7,141 0 20 Apr 2018
code2vec: Learning Distributed Representations of Code Uri Alon Meital Zilberstein Omer Levy Eran Yahav 53 1,171 0 26 Mar 2018
Deep contextualized word representations Matthew E. Peters Mark Neumann Mohit Iyyer Matt Gardner Christopher Clark Kenton Lee Luke Zettlemoyer NAI 186 11,542 0 15 Feb 2018
Mixed Precision Training Paulius Micikevicius Sharan Narang Jonah Alben G. Diamos Erich Elsen ... Boris Ginsburg Michael Houston Oleksii Kuchaiev Ganesh Venkatesh Hao Wu 149 1,792 0 10 Oct 2017
Sentiment Polarity Detection for Software Development Fabio Calefato F. Lanubile Federico Maiorano Nicole Novielli 31 228 0 09 Sep 2017
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 636 130,942 0 12 Jun 2017
SQuAD: 100,000+ Questions for Machine Comprehension of Text Pranav Rajpurkar Jian Zhang Konstantin Lopyrev Percy Liang RALM 235 8,113 0 16 Jun 2016
Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis A. Benavoli Giorgio Corani J. Demšar Marco Zaffalon BDL 62 422 0 14 Jun 2016
Characterizing Diseases from Unstructured Text: A Vocabulary Driven Word2vec Approach Saurav Ghosh Prithwish Chakraborty E. Cohn J. Brownstein Naren Ramakrishnan 40 33 0 01 Mar 2016
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen G. Corrado J. Dean 3DV 629 31,469 0 16 Jan 2013