Title
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness Tri Dao Daniel Y. Fu Stefano Ermon Atri Rudra Christopher Ré VLM 256 2,285 0 27 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 362 3,699 0 02 May 2022
PaLM: Scaling Language Modeling with Pathways Aakanksha Chowdhery Sharan Narang Jacob Devlin Maarten Bosma Gaurav Mishra ... Kathy Meier-Hellstern Douglas Eck J. Dean Slav Petrov Noah Fiedel PILM LRM 535 6,301 0 05 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
LaMDA: Language Models for Dialog Applications R. Thoppilan Daniel De Freitas Jamie Hall Noam M. Shazeer Apoorv Kulshreshtha ... Blaise Aguera-Arcas Claire Cui M. Croak Ed H. Chi Quoc Le ALM 146 1,601 0 20 Jan 2022
Pretrained Language Models for Text Generation: A Survey Junyi Li Tianyi Tang Wayne Xin Zhao J. Nie Ji-Rong Wen AI4CE 108 147 0 14 Jan 2022
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation Shuohuan Wang Yu Sun Yang Xiang Zhihua Wu Siyu Ding ... Tian Wu Wei Zeng Ge Li Wen Gao Haifeng Wang ELM 86 78 0 23 Dec 2021
GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin ... Kun Zhang Quoc V. Le Yonghui Wu Zhiwen Chen Claire Cui ALM MoE 233 829 0 13 Dec 2021
Efficiently Modeling Long Sequences with Structured State Spaces Albert Gu Karan Goel Christopher Ré 217 1,829 0 31 Oct 2021
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 355 1,709 0 15 Oct 2021
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation Yu Sun Shuohuan Wang Shikun Feng Siyu Ding Chao Pang ... Ouyang Xuan Dianhai Yu Hao Tian Hua Wu Haifeng Wang 112 472 0 05 Jul 2021
An Attention Free Transformer Shuangfei Zhai Walter A. Talbott Nitish Srivastava Chen Huang Hanlin Goh Ruixiang Zhang J. Susskind ViT 86 132 0 28 May 2021
PanGu- $α$ : Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation Wei Zeng Xiaozhe Ren Teng Su Hui Wang Yi-Lun Liao ... Gaojun Fan Yaowei Wang Xuefeng Jin Qun Liu Yonghong Tian ALM MoE AI4CE 78 213 0 26 Apr 2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Zhengxiao Du Yujie Qian Xiao Liu Ming Ding J. Qiu Zhilin Yang Jie Tang BDL AI4CE 151 1,556 0 18 Mar 2021
Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity W. Fedus Barret Zoph Noam M. Shazeer MoE 88 2,226 0 11 Jan 2021
SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning Hanrui Wang Zhekai Zhang Song Han 135 395 0 17 Dec 2020
mT5: A massively multilingual pre-trained text-to-text transformer Linting Xue Noah Constant Adam Roberts Mihir Kale Rami Al-Rfou Aditya Siddhant Aditya Barua Colin Raffel 148 2,560 0 22 Oct 2020
Big Bird: Transformers for Longer Sequences Manzil Zaheer Guru Guruganesh Kumar Avinava Dubey Joshua Ainslie Chris Alberti ... Philip Pham Anirudh Ravula Qifan Wang Li Yang Amr Ahmed VLM 568 2,103 0 28 Jul 2020
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention Angelos Katharopoulos Apoorv Vyas Nikolaos Pappas Franccois Fleuret 209 1,793 0 29 Jun 2020
DeBERTa: Decoding-enhanced BERT with Disentangled Attention Pengcheng He Xiaodong Liu Jianfeng Gao Weizhu Chen AAML 169 2,754 0 05 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 898 42,463 0 28 May 2020
Longformer: The Long-Document Transformer Iz Beltagy Matthew E. Peters Arman Cohan RALM VLM 187 4,100 0 10 Apr 2020
Sparse Sinkhorn Attention Yi Tay Dara Bahri Liu Yang Donald Metzler Da-Cheng Juan 91 342 0 26 Feb 2020
GLU Variants Improve Transformer Noam M. Shazeer 154 1,022 0 12 Feb 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 651 4,921 0 23 Jan 2020
Reformer: The Efficient Transformer Nikita Kitaev Lukasz Kaiser Anselm Levskaya VLM 207 2,333 0 13 Jan 2020
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension M. Lewis Yinhan Liu Naman Goyal Marjan Ghazvininejad Abdel-rahman Mohamed Omer Levy Veselin Stoyanov Luke Zettlemoyer AIMat VLM 266 10,861 0 29 Oct 2019
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer Colin Raffel Noam M. Shazeer Adam Roberts Katherine Lee Sharan Narang Michael Matena Yanqi Zhou Wei Li Peter J. Liu AIMat 500 20,342 0 23 Oct 2019
Adaptively Sparse Transformers Gonçalo M. Correia Vlad Niculae André F. T. Martins 98 256 0 30 Aug 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 697 24,557 0 26 Jul 2019
Generating Long Sequences with Sparse Transformers R. Child Scott Gray Alec Radford Ilya Sutskever 135 1,919 0 23 Apr 2019
The Curious Case of Neural Text Degeneration Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi 207 3,211 0 22 Apr 2019
Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context Zihang Dai Zhilin Yang Yiming Yang J. Carbonell Quoc V. Le Ruslan Salakhutdinov VLM 260 3,747 0 09 Jan 2019
Blockwise Parallel Decoding for Deep Autoregressive Models Mitchell Stern Noam M. Shazeer Ashley J. Llorens 70 238 0 07 Nov 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 805 132,725 0 12 Jun 2017
Gaussian Error Linear Units (GELUs) Dan Hendrycks Kevin Gimpel 174 5,049 0 27 Jun 2016