How Much Does Tokenization Affect Neural Machine Translation?

20 December 2018

Papers citing "How Much Does Tokenization Affect Neural Machine Translation?"

7 / 7 papers shown

Title
Low-Resourced Machine Translation for Senegalese Wolof Language Derguene Mbaye Moussa Diallo T. Diop 27 4 0 01 May 2023
How Robust is Neural Machine Translation to Language Imbalance in Multilingual Tokenizer Training? Shiyue Zhang Vishrav Chaudhary Naman Goyal James Cross Guillaume Wenzek Joey Tianyi Zhou Francisco Guzman 36 16 0 29 Apr 2022
Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP Sabrina J. Mielke Zaid Alyafeai Elizabeth Salesky Colin Raffel Manan Dey ... Arun Raja Chenglei Si Wilson Y. Lee Benoît Sagot Samson Tan 32 142 0 20 Dec 2021
Byte Pair Encoding is Suboptimal for Language Model Pretraining Kaj Bostrom Greg Durrett 14 200 0 07 Apr 2020
Neural Machine Translation: A Review and Survey Felix Stahlberg 3DV AI4TS MedIm 20 311 0 04 Dec 2019
OpenNMT: Open-Source Toolkit for Neural Machine Translation Guillaume Klein Yoon Kim Yuntian Deng Jean Senellart Alexander M. Rush 273 1,896 0 10 Jan 2017
Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation Yonghui Wu M. Schuster Z. Chen Quoc V. Le Mohammad Norouzi ... Alex Rudnick Oriol Vinyals G. Corrado Macduff Hughes J. Dean AIMat 716 6,746 0 26 Sep 2016