Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages

13 September 2021

Papers citing "Wine is Not v i n. -- On the Compatibility of Tokenizations Across Languages"

8 / 8 papers shown

Title
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation Francois Meyer Jan Buys 90 1 0 29 Mar 2024
Analyzing Cognitive Plausibility of Subword Tokenization Lisa Beinborn Yuval Pinter 63 20 0 20 Oct 2023
Differential Privacy, Linguistic Fairness, and Training Data Influence: Impossibility and Possibility Theorems for Multilingual Language Models Phillip Rust Anders Søgaard 68 3 0 17 Aug 2023
A Multi-dimensional Evaluation of Tokenizer-free Multilingual Pretrained Models Jimin Sun Patrick Fernandes Xinyi Wang Graham Neubig 100 12 0 13 Oct 2022
Incorporating Context into Subword Vocabularies Shaked Yehezkel Yuval Pinter 100 10 0 13 Oct 2022
Language Modelling with Pixels Phillip Rust Jonas F. Lotz Emanuele Bugliarello Elizabeth Salesky Miryam de Lhoneux Desmond Elliott VLM 107 46 0 14 Jul 2022
Overlap-based Vocabulary Generation Improves Cross-lingual Transfer Among Related Languages Vaidehi Patil Partha P. Talukdar Sunita Sarawagi 122 22 0 03 Mar 2022
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP Sabrina J. Mielke Zaid Alyafeai Elizabeth Salesky Colin Raffel Manan Dey ... Arun Raja Chenglei Si Wilson Y. Lee Benoît Sagot Samson Tan 109 151 0 20 Dec 2021