SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,950 papers shown

Title
Two to Five Truths in Non-Negative Matrix Factorization John M. Conroy Neil P. Molino Brian Baughman Rod Gomez Ryan Kaliszewski Nicholas A. Lines 74 0 0 06 May 2023
Pre-training Language Model as a Multi-perspective Course Learner Beiduo Chen Shaohan Huang Zi-qiang Zhang Wu Guo Zhen-Hua Ling Haizhen Huang Furu Wei Weiwei Deng Qi Zhang 58 0 0 06 May 2023
Now It Sounds Like You: Learning Personalized Vocabulary On Device Sida Wang Ashish Shenoy P. Chuang John Nguyen VLM 91 3 0 05 May 2023
Investigating Lexical Sharing in Multilingual Machine Translation for Indian Languages Sonal Sannigrahi Rachel Bawden 57 0 0 04 May 2023
Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks Yun Tang Anna Y. Sun Hirofumi Inaguma Xinyue Chen Ning Dong Xutai Ma Paden Tomasello J. Pino 108 22 0 04 May 2023
What changes when you randomly choose BPE merge operations? Not much Jonne Saleva Constantine Lignos 53 7 0 04 May 2023
Learning Language-Specific Layers for Multilingual Machine Translation Telmo Pires Robin M. Schmidt Yi-Hsiu Liao Stephan Peitz 109 19 0 04 May 2023
Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity Da Xu Maha Elbayad Kenton W. Murray Jean Maillard Vedanuj Goswami MoE 62 3 0 03 May 2023
Low-Resourced Machine Translation for Senegalese Wolof Language Derguene Mbaye Moussa Diallo T. Diop 59 4 0 01 May 2023
ResiDual: Transformer with Dual Residual Connections Shufang Xie Huishuai Zhang Junliang Guo Xu Tan Jiang Bian Hany Awadalla Arul Menezes Tao Qin Rui Yan 99 19 0 28 Apr 2023
Training and Evaluation of a Multilingual Tokenizer for GPT-SW3 Felix Stollenwerk 83 8 0 28 Apr 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 311 956 0 27 Apr 2023
Semantic Tokenizer for Enhanced Natural Language Processing Sandeep Mehta Darpan Shah Ravindra Kulkarni Cornelia Caragea VLM 26 3 0 24 Apr 2023
NAIST-SIC-Aligned: an Aligned English-Japanese Simultaneous Interpretation Corpus Jinming Zhao Yuka Ko Kosuke Doi Ryo Fukuda Katsuhito Sudoh Satoshi Nakamura 152 2 0 23 Apr 2023
Tokenization Preference for Human and Machine Learning Model: An Annotation Study Tatsuya Hiraoka Tomoya Iwakura 51 1 0 21 Apr 2023
Downstream Task-Oriented Neural Tokenizer Optimization with Vocabulary Restriction as Post Processing Tatsuya Hiraoka Tomoya Iwakura 66 0 0 21 Apr 2023
Joint Repetition Suppression and Content Moderation of Large Language Models Minghui Zhang Alex Sokolov Weixin Cai Si-Qing Chen 43 1 0 20 Apr 2023
MPMQA: Multimodal Question Answering on Product Manuals Liangfu Zhang Anwen Hu Jing Zhang Shuo Hu Qin Jin 84 10 0 19 Apr 2023
UniMax: Fairer and more Effective Language Sampling for Large-Scale Multilingual Pretraining Hyung Won Chung Noah Constant Xavier Garcia Adam Roberts Yi Tay Sharan Narang Orhan Firat 116 57 0 18 Apr 2023
From Words to Music: A Study of Subword Tokenization Techniques in Symbolic Music Generation Adarsh Kumar Pedro Sarmento 69 4 0 18 Apr 2023
Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese Vésteinn Snaebjarnarson A. Simonsen Goran Glavaš Ivan Vulić 79 23 0 18 Apr 2023
A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models Qianqian Xie Zheheng Luo Benyou Wang Sophia Ananiadou LM&MA VLM 53 11 0 18 Apr 2023
The MiniPile Challenge for Data-Efficient Language Models Jean Kaddour MoE ALM 133 46 0 17 Apr 2023
VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning Zhen-Ru Zhang Chuanqi Tan Songfang Huang Fei Huang VLM 64 5 0 17 Apr 2023
Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca Yiming Cui Ziqing Yang Xin Yao ALM 96 317 0 17 Apr 2023
Neural Machine Translation For Low Resource Languages V. Goyle Parvathy Krishnaswamy K. G. Ravikumar Utsa Chattopadhyay Kartikay Goyle 29 0 0 16 Apr 2023
Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation Yunjie Ji Yan Gong Yong Deng Yiping Peng Qiang Niu Baochang Ma Xiangang Li ALM ELM 102 25 0 16 Apr 2023
A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition Ruchao Fan Wei Chu Peng Chang Abeer Alwan 36 11 0 15 Apr 2023
Sign Language Translation from Instructional Videos Laia Tarrés Gerard I. Gállego A. Duarte Jordi Torres Xavier Giró-i-Nieto SLR 75 34 0 13 Apr 2023
Computational modeling of semantic change Nina Tahmasebi Haim Dubossarsky 104 6 0 13 Apr 2023
Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis Wenhao Zhu Hongyi Liu Qingxiu Dong Jingjing Xu Shujian Huang Lingpeng Kong Jiajun Chen Lei Li LRM 131 152 0 10 Apr 2023
PEACH: Pre-Training Sequence-to-Sequence Multilingual Models for Translation with Semi-Supervised Pseudo-Parallel Document Generation Alireza Salemi Amirhossein Abaskohi Sara Tavakoli Yadollah Yaghoobzadeh A. Shakery AIMat 76 0 0 03 Apr 2023
DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains Yanis Labrak Adrien Bazoge Richard Dufour Mickael Rouvier Emmanuel Morin B. Daille P. Gourraud LM&MA 71 57 0 03 Apr 2023
GreekBART: The First Pretrained Greek Sequence-to-Sequence Model Iakovos Evdaimon Hadi Abdine Christos Xypolopoulos Stamatis Outsios Michalis Vazirgiannis Giorgos Stamou VLM 42 7 0 03 Apr 2023
Spam-T5: Benchmarking Large Language Models for Few-Shot Email Spam Detection Maxime Labonne Sean J. Moran 90 23 0 03 Apr 2023
Exploiting Multilingualism in Low-resource Neural Machine Translation via Adversarial Learning Amit Kumar A. Pratap Anil Kumar Singh AI4CE 58 2 0 31 Mar 2023
BloombergGPT: A Large Language Model for Finance Shijie Wu Ozan Irsoy Steven Lu Vadim Dabravolski Mark Dredze Sebastian Gehrmann P. Kambadur David S. Rosenberg Gideon Mann AIFin 246 853 0 30 Mar 2023
A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision Lucas Beyer Bo Wan Gagan Madan Filip Pavetić Andreas Steiner ... Emanuele Bugliarello Tianlin Li Qihang Yu Liang-Chieh Chen Xiaohua Zhai 130 9 0 30 Mar 2023
TreePiece: Faster Semantic Parsing via Tree Tokenization Sida I. Wang Akshat Shrivastava S. Livshits 61 5 0 30 Mar 2023
When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP Sara Papi Marco Gaido Andrea Pilzer Matteo Negri 115 12 0 28 Mar 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 298 1,206 0 27 Mar 2023
Cross-utterance ASR Rescoring with Graph-based Label Propagation Srinath Tankasala Long Chen A. Stolcke A. Raju Qianli Deng Chander Chandak Aparna Khare Roland Maas Venkatesh Ravichandran 48 0 0 27 Mar 2023
An Information Extraction Study: Take In Mind the Tokenization! Christos Theodoropoulos Marie-Francine Moens 45 6 0 27 Mar 2023
Fine-Tashkeel: Finetuning Byte-Level Models for Accurate Arabic Text Diacritization Bashar Al-Rfooh Gheith A. Abandah Rami Al-Rfou 54 7 0 25 Mar 2023
Neuro-Symbolic Execution of Generic Source Code Yaojie Hu Jin Tian NAI 85 0 0 23 Mar 2023
SwissBERT: The Multilingual Language Model for Switzerland Jannis Vamvas Johannes Graen Rico Sennrich 81 7 0 23 Mar 2023
A Gold Standard Dataset for the Reviewer Assignment Problem Ivan Stelmakh John Wieting Sarina Xi Graham Neubig Nihar B. Shah 112 16 0 23 Mar 2023
JaCoText: A Pretrained Model for Java Code-Text Generation Jessica Nayeli López Espejel Mahaman Sanoussi Yahaya Alassan Walid Dahhane E. Ettifouri 56 4 0 22 Mar 2023
Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition Xiaoyu Yang Qiujia Li Chuxu Zhang P. Woodland 85 7 0 20 Mar 2023
Character, Word, or Both? Revisiting the Segmentation Granularity for Chinese Pre-trained Language Models Xinnian Liang Zefan Zhou Hui Huang Shuangzhi Wu Tong Xiao Muyun Yang Zhoujun Li Chao Bian VLM 65 2 0 20 Mar 2023