Title
Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation Thomas F Burns Letitia Parcalabescu Stephan Wäldchen Michael Barlow Gregor Ziegltrum Volker Stampa Bastian Harren Björn Deiseroth SyDa 121 0 0 24 Apr 2025
LLäMmlein: Transparent, Compact and Competitive German-Only Language Models from Scratch Jan Pfister Julia Wunderle Andreas Hotho 89 2 0 17 Nov 2024
Smirk: An Atomically Complete Tokenizer for Molecular Foundation Models Alexius Wadell Anoushka Bhutani Venkatasubramanian Viswanathan 451 1 0 19 Sep 2024
On the Challenges and Opportunities in Generative AI Laura Manduchi Kushagra Pandey Robert Bamler Ryan Cotterell Sina Daubener ... F. Wenzel Frank Wood Stephan Mandt Vincent Fortuin Vincent Fortuin 276 22 0 28 Feb 2024
GPT-SW3: An Autoregressive Language Model for the Nordic Languages Ariel Ekgren Amaru Cuba Gyllensten Felix Stollenwerk Joey Öhman T. Isbister Evangelia Gogoulou F. Carlsson Alice Heiman Judit Casademont Magnus Sahlgren 83 13 0 22 May 2023
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers L. Yu Daniel Simig Colin Flaherty Armen Aghajanyan Luke Zettlemoyer M. Lewis 70 93 0 12 May 2023
Training and Evaluation of a Multilingual Tokenizer for GPT-SW3 Felix Stollenwerk 75 8 0 28 Apr 2023
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model BigScience Workshop : Teven Le Scao Angela Fan Christopher Akiki ... Zhongli Xie Zifan Ye M. Bras Younes Belkada Thomas Wolf VLM 414 2,393 0 09 Nov 2022
Impact of Tokenization on Language Models: An Analysis for Turkish Cagri Toraman E. Yilmaz Furkan Şahinuç Oguzhan Ozcelik 99 81 0 19 Apr 2022
mGPT: Few-Shot Learners Go Multilingual Oleh Shliazhko Alena Fenogenova Maria Tikhonova Vladislav Mikhailov Anastasia Kozlova Tatiana Shavrina 90 154 0 15 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 208 1,987 0 29 Mar 2022
Few-shot Learning with Multilingual Language Models Xi Lin Todor Mihaylov Mikel Artetxe Tianlu Wang Shuohui Chen ... Luke Zettlemoyer Zornitsa Kozareva Mona T. Diab Ves Stoyanov Xian Li BDL ELM LRM 133 309 0 20 Dec 2021
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization Yi Tay Vinh Q. Tran Sebastian Ruder Jai Gupta Hyung Won Chung Dara Bahri Zhen Qin Simon Baumgartner Cong Yu Donald Metzler 135 160 0 23 Jun 2021
ByT5: Towards a token-free future with pre-trained byte-to-byte models Linting Xue Aditya Barua Noah Constant Rami Al-Rfou Sharan Narang Mihir Kale Adam Roberts Colin Raffel 104 506 0 28 May 2021
Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Deepak Narayanan Mohammad Shoeybi Jared Casper P. LeGresley M. Patwary ... Prethvi Kashinkunti J. Bernauer Bryan Catanzaro Amar Phanishayee Matei A. Zaharia MoE 140 707 0 09 Apr 2021
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation J. Clark Dan Garrette Iulia Turc John Wieting 100 224 0 11 Mar 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 476 2,123 0 31 Dec 2020
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models Phillip Rust Jonas Pfeiffer Ivan Vulić Sebastian Ruder Iryna Gurevych 137 255 0 31 Dec 2020
Morphology Matters: A Multilingual Language Modeling Analysis Hyunji Hayley Park Katherine J. Zhang Coleman Haley K. Steimel Han Liu Lane Schwartz 98 49 0 11 Dec 2020
XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning Edoardo Ponti Goran Glavaš Olga Majewska Qianchu Liu Ivan Vulić Anna Korhonen LRM 98 328 0 01 May 2020
Byte Pair Encoding is Suboptimal for Language Model Pretraining Kaj Bostrom Greg Durrett 65 212 0 07 Apr 2020
PIQA: Reasoning about Physical Commonsense in Natural Language Yonatan Bisk Rowan Zellers Ronan Le Bras Jianfeng Gao Yejin Choi OOD LRM 189 1,847 0 26 Nov 2019
PubMedQA: A Dataset for Biomedical Research Question Answering Qiao Jin Bhuwan Dhingra Zhengping Liu William W. Cohen Xinghua Lu 401 914 0 13 Sep 2019
Neural Machine Translation with Byte-Level Subwords Changhan Wang Kyunghyun Cho Jiatao Gu 87 178 0 07 Sep 2019
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions Christopher Clark Kenton Lee Ming-Wei Chang Tom Kwiatkowski Michael Collins Kristina Toutanova 247 1,562 0 24 May 2019
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations Mohammad Taher Pilehvar Jose Camacho-Collados 207 492 0 28 Aug 2018
SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing Taku Kudo John Richardson 209 3,534 0 19 Aug 2018
Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates Taku Kudo 232 1,173 0 29 Apr 2018
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding Alex Jinpeng Wang Amanpreet Singh Julian Michael Felix Hill Omer Levy Samuel R. Bowman ELM 1.1K 7,201 0 20 Apr 2018
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension Mandar Joshi Eunsol Choi Daniel S. Weld Luke Zettlemoyer RALM 237 2,692 0 09 May 2017
A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference Adina Williams Nikita Nangia Samuel R. Bowman 524 4,497 0 18 Apr 2017
RACE: Large-scale ReAding Comprehension Dataset From Examinations Guokun Lai Qizhe Xie Hanxiao Liu Yiming Yang Eduard H. Hovy ELM 203 1,359 0 15 Apr 2017
Neural Machine Translation of Rare Words with Subword Units Rico Sennrich Barry Haddow Alexandra Birch 238 7,765 0 31 Aug 2015