SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,950 papers shown

Title
HIT-SCIR at MMNLU-22: Consistency Regularization for Multilingual Spoken Language Understanding Bo Zheng Zhouyang Li Fuxuan Wei Qiguang Chen Libo Qin Wanxiang Che 47 4 0 05 Jan 2023
Audio-Visual Efficient Conformer for Robust Speech Recognition Maxime Burchi Radu Timofte VLM 78 35 0 04 Jan 2023
Cramming: Training a Language Model on a Single GPU in One Day Jonas Geiping Tom Goldstein MoE 117 91 0 28 Dec 2022
Optimizing Deep Transformers for Chinese-Thai Low-Resource Translation Wenjie Hao Hongfei Xu Lingling Mu Hongying Zan MoE 97 4 0 24 Dec 2022
Pushing the performances of ASR models on English and Spanish accents Pooja Chitkara M. Rivière Jade Copet Frank Zhang Yatharth Saraf 46 0 0 22 Dec 2022
Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models Najoung Kim Tal Linzen P. Smolensky 106 33 0 21 Dec 2022
ORCA: A Challenging Benchmark for Arabic Language Understanding AbdelRahim Elmadany El Moatez Billah Nagoudi Muhammad Abdul-Mageed ELM 109 45 0 21 Dec 2022
Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval John Wieting J. Clark William W. Cohen Graham Neubig Taylor Berg-Kirkpatrick 100 6 0 21 Dec 2022
Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Training Kelly Marchisio Patrick Lewis Yihong Chen Mikel Artetxe 80 19 0 20 Dec 2022
ByGPT5: End-to-End Style-conditioned Poetry Generation with Token-free Language Models Jonas Belouadi Steffen Eger 101 26 0 20 Dec 2022
Little Red Riding Hood Goes Around the Globe:Crosslingual Story Planning and Generation with Large Language Models E. Razumovskaia Joshua Maynez Annie Louis Mirella Lapata Shashi Narayan LRM 58 5 0 20 Dec 2022
SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers Hongyi Yuan Zheng Yuan Chuanqi Tan Fei Huang Songfang Huang DiffM 157 70 0 20 Dec 2022
GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator Jian Yang Shuming Ma Li Dong Shaohan Huang Haoyang Huang Yuwei Yin Dongdong Zhang Liqun Yang Furu Wei Zhoujun Li SyDa AI4CE 76 25 0 20 Dec 2022
A Survey on Pretrained Language Models for Neural Code Intelligence Yichen Xu Yanqiao Zhu 52 17 0 20 Dec 2022
Joint Speech Transcription and Translation: Pseudo-Labeling with Out-of-Distribution Data Mozhdeh Gheini Tatiana Likhomanenko Matthias Sperber Hendra Setiawan 90 5 0 20 Dec 2022
Tokenization Consistency Matters for Generative Models on Extractive NLP Tasks Kaiser Sun Peng Qi Yuhao Zhang Lan Liu William Yang Wang Zhiheng Huang 80 9 0 19 Dec 2022
Synthetic Pre-Training Tasks for Neural Machine Translation Zexue He Graeme W. Blackwood Yikang Shen Julian McAuley Rogerio Feris 54 4 0 19 Dec 2022
(Psycho-)Linguistic Features Meet Transformer Models for Improved Explainable and Controllable Text Simplification Yu Qiao Xiaofei Li Daniel Wiechmann E. Kerz 55 4 0 19 Dec 2022
SegAugment: Maximizing the Utility of Speech Translation Data with Segmentation-based Augmentations Ioannis Tsiamas José A. R. Fonollosa Marta R. Costa-jussá 84 6 0 19 Dec 2022
A Natural Bias for Language Generation Models Clara Meister Wojciech Stokowiec Tiago Pimentel Lei Yu Laura Rimell A. Kuncoro MILM 89 6 0 19 Dec 2022
Large Language Models Meet NL2Code: A Survey Daoguang Zan B. Chen Fengji Zhang Di Lu Bingchao Wu Bei Guan Yongji Wang Jian-Guang Lou ELM ALM 95 183 0 19 Dec 2022
WACO: Word-Aligned Contrastive Learning for Speech Translation Siqi Ouyang Rong Ye Lei Li 104 28 0 19 Dec 2022
AdaTranS: Adapting with Boundary-based Shrinking for End-to-End Speech Translation Xingshan Zeng Liangyou Li Qun Liu 59 5 0 17 Dec 2022
Controlling Styles in Neural Machine Translation with Activation Prompt Yifan Wang Zewei Sun Shanbo Cheng Weiguo Zheng Mingxuan Wang 87 10 0 17 Dec 2022
Planting and Mitigating Memorized Content in Predictive-Text Language Models C.M. Downey Wei Dai Huseyin A. Inan Kim Laine Saurabh Naik Tomasz Religa PILM 34 2 0 16 Dec 2022
UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units Hirofumi Inaguma Sravya Popuri Ilia Kulikov Peng-Jen Chen Changhan Wang Yu-An Chung Yun Tang Ann Lee Shinji Watanabe J. Pino 119 61 0 15 Dec 2022
CLIPPO: Image-and-Language Understanding from Pixels Only Michael Tschannen Basil Mustafa N. Houlsby CLIP VLM 102 49 0 15 Dec 2022
Advancing Multilingual Pre-training: TRIP Triangular Document-level Pre-training for Multilingual Language Models Hongyuan Lu Haoyang Huang Shuming Ma Dongdong Zhang W. Lam Furu Wei 67 4 0 15 Dec 2022
Fixing MoE Over-Fitting on Low-Resource Languages in Multilingual Machine Translation Maha Elbayad Anna Y. Sun Shruti Bhosale MoE 84 10 0 15 Dec 2022
Causes and Cures for Interference in Multilingual Translation Uri Shaham Maha Elbayad Vedanuj Goswami Omer Levy Shruti Bhosale 97 26 0 14 Dec 2022
ERNIE-Code: Beyond English-Centric Cross-lingual Pretraining for Programming Languages Yekun Chai Shuohuan Wang Chao Pang Yu Sun Hao Tian Hua Wu 93 38 0 13 Dec 2022
Jointly Learning Visual and Auditory Speech Representations from Raw Data A. Haliassos Pingchuan Ma Rodrigo Mira Stavros Petridis Maja Pantic SSL 92 49 0 12 Dec 2022
M3ST: Mix at Three Levels for Speech Translation Xuxin Cheng Qianqian Dong Fengpeng Yue Tom Ko Mingxuan Wang Yuexian Zou 96 41 0 07 Dec 2022
Rethinking the Objectives of Vector-Quantized Tokenizers for Image Synthesis Yuchao Gu Xintao Wang Yixiao Ge Ying Shan Xiaohu Qie Mike Zheng Shou DiffM 98 22 0 06 Dec 2022
Document-Level Abstractive Summarization Gonçalo Raposo Afonso Raposo Ana Sofia Carmo 48 2 0 06 Dec 2022
LMEC: Learnable Multiplicative Absolute Position Embedding Based Conformer for Speech Recognition Yuguang Yang Yu Pan Jingjing Yin Heng Lu 100 3 0 05 Dec 2022
Democratizing Neural Machine Translation with OPUS-MT Jörg Tiedemann Mikko Aulamo Daria Bakshandaeva M. Boggia Stig-Arne Gronroos Tommi Nieminen Alessandro Raganato Yves Scherrer Raúl Vázquez Sami Virpioja 122 32 0 04 Dec 2022
Unsupervised Fine-Tuning Data Selection for ASR Using Self-Supervised Speech Models Reem Gody David Harwath 47 4 0 03 Dec 2022
Improving Simultaneous Machine Translation with Monolingual Data Hexuan Deng Liang Ding Xuebo Liu Meishan Zhang Dacheng Tao Min Zhang 93 15 0 02 Dec 2022
Tackling Low-Resourced Sign Language Translation: UPC at WMT-SLT 22 Laia Tarrés Gerard I. Gállego Xavier Giró-i-Nieto Jordi Torres SLR 81 5 0 02 Dec 2022
CUNI Systems for the WMT22 Czech-Ukrainian Translation Task Martin Popel Jindrich Libovický Jindřich Helcl 46 4 0 01 Dec 2022
Exploiting Category Names for Few-Shot Classification with Vision-Language Models Taihong Xiao Zirui Wang Liangliang Cao Jiahui Yu Shengyang Dai Ming-Hsuan Yang VLM MLLM 87 5 0 29 Nov 2022
BARTSmiles: Generative Masked Language Models for Molecular Representations Gayane Chilingaryan Hovhannes Tamoyan Ani Tevosyan N. Babayan L. Khondkaryan Karen Hambardzumyan Zaven Navoyan Hrant Khachatrian Armen Aghajanyan SSL 101 28 0 29 Nov 2022
CUNI Submission in WMT22 General Task Josef Jon Martin Popel Ondrej Bojar 50 6 0 29 Nov 2022
Extending the Subwording Model of Multilingual Pretrained Models for New Languages K. Imamura Eiichiro Sumita VLM 83 3 0 29 Nov 2022
Continuous diffusion for categorical data Sander Dieleman Laurent Sartran Arman Roshannai Nikolay Savinov Yaroslav Ganin ... Conor Durkan Curtis Hawthorne Rémi Leblond Will Grathwohl J. Adler DiffM 121 106 0 28 Nov 2022
Mask the Correct Tokens: An Embarrassingly Simple Approach for Error Correction Kai Shen Yichong Leng Xuejiao Tan Si-Qi Tang Yuan Zhang Wenjie Liu Ed Lin 69 15 0 23 Nov 2022
Benchmarking Evaluation Metrics for Code-Switching Automatic Speech Recognition Injy Hamed A. Hussein Oumnia Chellah Shammur A. Chowdhury Hamdy Mubarak Sunayana Sitaram Nizar Habash Ahmed M. Ali 83 6 0 22 Nov 2022
VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learning Qiu-shi Zhu Long Zhou Zi-Hua Zhang Shujie Liu Binxing Jiao Jie Zhang Lirong Dai Daxin Jiang Jinyu Li Furu Wei 113 38 0 21 Nov 2022
Efficient Transformers with Dynamic Token Pooling Piotr Nawrot J. Chorowski Adrian Lañcucki Edoardo Ponti 83 46 0 17 Nov 2022