Improving Multilingual Models with Language-Clustered Vocabularies

24 October 2020

Papers citing "Improving Multilingual Models with Language-Clustered Vocabularies"

29 / 29 papers shown

Title
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? HyoJung Han Akiko Eriguchi Haoran Xu Hieu T. Hoang Marine Carpuat Huda Khayrallah VLM 34 2 0 12 Oct 2024
DEPT: Decoupled Embeddings for Pre-training Language Models Alex Iacob Lorenzo Sani Meghdad Kurmanji William F. Shen Xinchi Qiu Dongqi Cai Yan Gao Nicholas D. Lane VLM 145 0 0 07 Oct 2024
SUTRA: Scalable Multilingual Language Model Architecture Abhijit Bendale Michael Sapienza Steven Ripplinger Simon Gibbs Jaewon Lee Pranav Mistry LRM ELM 36 4 0 07 May 2024
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation Francois Meyer Jan Buys 33 1 0 29 Mar 2024
How Vocabulary Sharing Facilitates Multilingualism in LLaMA? Fei Yuan Shuai Yuan Zhiyong Wu Lei Li 31 10 0 15 Nov 2023
Analyzing Cognitive Plausibility of Subword Tokenization Lisa Beinborn Yuval Pinter 29 17 0 20 Oct 2023
CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code Nadezhda Chirkova Sergey Troshin 21 8 0 01 Aug 2023
An Efficient Multilingual Language Model Compression through Vocabulary Trimming Asahi Ushio Yi Zhou Jose Camacho-Collados 41 7 0 24 May 2023
Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models Orevaoghene Ahia Sachin Kumar Hila Gonen Jungo Kasai David R. Mortensen Noah A. Smith Yulia Tsvetkov 40 80 0 23 May 2023
Glot500: Scaling Multilingual Corpora and Language Models to 500 Languages Ayyoob Imani Peiqin Lin Amir Hossein Kargaran Silvia Severini Masoud Jalili Sabet ... Chunlan Ma Helmut Schmid André F. T. Martins François Yvon Hinrich Schütze ALM LRM 31 95 0 20 May 2023
Language Model Tokenizers Introduce Unfairness Between Languages Aleksandar Petrov Emanuele La Malfa Philip H. S. Torr Adel Bibi 16 96 0 17 May 2023
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining Hyung Won Chung Noah Constant Xavier Garcia Adam Roberts Yi Tay Sharan Narang Orhan Firat 23 49 0 18 Apr 2023
Incorporating Context into Subword Vocabularies Shaked Yehezkel Yuval Pinter 41 8 0 13 Oct 2022
Cross-Lingual Knowledge Transfer for Clinical Phenotyping Jens-Michalis Papaioannou Paul Grundmann Betty van Aken A. Samaras Ilias Kyparissidis G. Giannakoulas Felix Alexander Gers Alexander Loser 34 6 0 03 Aug 2022
Language Modelling with Pixels Phillip Rust Jonas F. Lotz Emanuele Bugliarello Elizabeth Salesky Miryam de Lhoneux Desmond Elliott VLM 35 46 0 14 Jul 2022
Feature Aggregation in Zero-Shot Cross-Lingual Transfer Using Multilingual BERT Beiduo Chen Wu Guo Quan Liu Kun Tao 35 1 0 17 May 2022
Lifting the Curse of Multilinguality by Pre-training Modular Transformers Jonas Pfeiffer Naman Goyal Xi Victoria Lin Xian Li James Cross Sebastian Riedel Mikel Artetxe LRM 40 139 0 12 May 2022
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training? Shiyue Zhang Vishrav Chaudhary Naman Goyal James Cross Guillaume Wenzek Mohit Bansal Francisco Guzman 31 16 0 29 Apr 2022
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP Sabrina J. Mielke Zaid Alyafeai Elizabeth Salesky Colin Raffel Manan Dey ... Arun Raja Chenglei Si Wilson Y. Lee Benoît Sagot Samson Tan 30 141 0 20 Dec 2021
Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training Bo Zheng Li Dong Shaohan Huang Saksham Singhal Wanxiang Che Ting Liu Xia Song Furu Wei VLM 13 22 0 15 Sep 2021
How Suitable Are Subword Segmentation Strategies for Translating Non-Concatenative Morphology? Chantal Amrhein Rico Sennrich 27 13 0 02 Sep 2021
A Primer on Pretrained Multilingual Language Models Sumanth Doddapaneni Gowtham Ramesh Mitesh M. Khapra Anoop Kunchukuttan Pratyush Kumar LRM 43 73 0 01 Jul 2021
Specializing Multilingual Language Models: An Empirical Study Ethan C. Chau Noah A. Smith 27 27 0 16 Jun 2021
Poolingformer: Long Document Modeling with Pooling Attention Hang Zhang Yeyun Gong Yelong Shen Weisheng Li Jiancheng Lv Nan Duan Weizhu Chen 35 98 0 10 May 2021
MuRIL: Multilingual Representations for Indian Languages Simran Khanuja Diksha Bansal Sarvesh Mehtani Savya Khosla Atreyee Dey ... Shachi Dave Shruti Gupta Subhash Chandra Bose Gali Vishnu Subramanian Partha P. Talukdar 41 277 0 19 Mar 2021
CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation J. Clark Dan Garrette Iulia Turc John Wieting 30 210 0 11 Mar 2021
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts Jonas Pfeiffer Ivan Vulić Iryna Gurevych Sebastian Ruder 24 126 0 31 Dec 2020
Scaling Laws for Neural Language Models Jared Kaplan Sam McCandlish T. Henighan Tom B. Brown B. Chess R. Child Scott Gray Alec Radford Jeff Wu Dario Amodei 240 4,469 0 23 Jan 2020
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,743 0 26 Sep 2016