SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,950 papers shown

Title
Revisiting Interpolation Augmentation for Speech-to-Text Generation Chen Xu Jie Wang Xiaoqian Liu Qianqian Dong Chunliang Zhang Tong Xiao Jingbo Zhu Dapeng Man Wu Yang 72 1 0 22 Jun 2024
TacoLM: GaTed Attention Equipped Codec Language Model are Efficient Zero-Shot Text to Speech Synthesizers Yakun Song Zhuo Chen Xiaofei Wang Ziyang Ma Guanrou Yang Xie Chen AuLLM 52 4 0 22 Jun 2024
Speech Prefix-Tuning with RNNT Loss for Improving LLM Predictions M. Baskar Andrew Rosenberg Bhuvana Ramabhadran Neeraj Gaur Zhong Meng 51 3 0 20 Jun 2024
Exploring Design Choices for Building Language-Specific LLMs Atula Tejaswi Nilesh Gupta Eunsol Choi 109 11 0 20 Jun 2024
How to Compute the Probability of a Word Tiago Pimentel Clara Meister 117 20 0 20 Jun 2024
Infusing clinical knowledge into tokenisers for language models Abul Hasan Jinge Wu Quang Ngoc Nguyen Salomé Andres Imane Guellil Huayu Zhang Arlene Casey Beatrice Alex Bruce Guthrie Honghan Wu 79 2 0 20 Jun 2024
On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations? Rochelle Choenni Sara Rajaee Christof Monz Ekaterina Shutova 78 2 0 20 Jun 2024
Lexically Grounded Subword Segmentation Jindřich Libovický Jindřich Helcl 114 3 0 19 Jun 2024
How effective is Multi-source pivoting for Translation of Low Resource Indian Languages? Pranav Gaikwad Meet Doshi Raj Dabre Pushpak Bhattacharyya 75 0 0 19 Jun 2024
Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? Jinhyuk Lee Anthony Chen Zhuyun Dai Dheeru Dua Devendra Singh Sachan ... Jeremy R. Cole Sebastian Riedel Iftekhar Naim Ming-Wei Chang Kelvin Guu RALM LRM 107 37 0 19 Jun 2024
Nemotron-4 340B Technical Report Nvidia : Bo Adler Niket Agarwal Ashwath Aithal ... Jimmy Zhang Jing Zhang Vivienne Zhang Yian Zhang Chen Zhu 121 69 0 17 Jun 2024
Tokenization Falling Short: The Curse of Tokenization Yekun Chai Yewei Fang Qiwei Peng Xuhong Li 74 1 0 17 Jun 2024
Towards an End-to-End Framework for Invasive Brain Signal Decoding with Large Language Models Sheng Feng Heyang Liu Yu Wang Yanfeng Wang 49 3 0 17 Jun 2024
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning Zebang Cheng Zhi-Qi Cheng Jun-Yan He Jingdong Sun Kai Wang Yuxiang Lin Zheng Lian Xiaojiang Peng Alexander G. Hauptmann MLLM 119 40 0 17 Jun 2024
Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation Boxuan Lyu Hidetaka Kamigaito Kotaro Funakoshi Manabu Okumura 170 0 0 17 Jun 2024
CoSTA: Code-Switched Speech Translation using Aligned Speech-Text Interleaving Bhavani Shankar Preethi Jyothi Pushpak Bhattacharyya 90 1 0 16 Jun 2024
Multilingual Large Language Models and Curse of Multilinguality Daniil Gurgurov Tanja Bäumel Tatiana Anikina 137 7 0 15 Jun 2024
CNVSRC 2023: The First Chinese Continuous Visual Speech Recognition Challenge Chen Chen Zehua Liu Xiaolou Li Lantian Li D. Wang 66 4 0 14 Jun 2024
UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languages Trinh Pham Khoi M. Le Luu Anh Tuan 116 1 0 14 Jun 2024
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities Roman Bachmann Oğuzhan Fatih Kar David Mizrahi Ali Garjani Mingfei Gao David Griffiths Jiaming Hu Afshin Dehghan Amir Zamir MoE VLM MLLM 106 17 0 13 Jun 2024
Transformer-based Model for ASR N-Best Rescoring and Rewriting Iwen E. Kang Christophe Van Gysel Man-Hung Siu 78 3 0 12 Jun 2024
An Empirical Study of Mamba-based Language Models R. Waleffe Wonmin Byeon Duncan Riach Brandon Norick V. Korthikanti ... Vartika Singh Jared Casper Jan Kautz Mohammad Shoeybi Bryan Catanzaro 125 78 0 12 Jun 2024
PRoDeliberation: Parallel Robust Deliberation for End-to-End Spoken Language Understanding Trang Le Daniel Lazar Suyoun Kim Shan Jiang Duc Le Adithya Sagar Aleksandr Livshits Ahmed Aly Akshat Shrivastava 76 0 0 12 Jun 2024
Languages Transferred Within the Encoder: On Representation Transfer in Zero-Shot Multilingual Translation Zhi Qu Chenchen Ding Taro Watanabe 161 1 0 12 Jun 2024
A Non-autoregressive Generation Framework for End-to-End Simultaneous Speech-to-Any Translation Zhengrui Ma Qingkai Fang Shaolei Zhang Shoutao Guo Yang Feng Min Zhang 83 11 0 11 Jun 2024
EAVE: Efficient Product Attribute Value Extraction via Lightweight Sparse-layer Interaction Li Yang Qifan Wang Jianfeng Chi Jiahao Liu Jingang Wang Fuli Feng Zenglin Xu Yi Fang Lifu Huang Dongfang Liu 75 1 0 10 Jun 2024
StreamAtt: Direct Streaming Speech-to-Text Translation with Attention-based Audio History Selection Sara Papi Marco Gaido Matteo Negri L. Bentivogli 139 8 0 10 Jun 2024
Attention as a Hypernetwork Simon Schug Seijin Kobayashi Yassir Akram João Sacramento Razvan Pascanu GNN 83 5 0 09 Jun 2024
Exploring the Benefits of Tokenization of Discrete Acoustic Units Avihu Dekel Raul Fernandez 78 2 0 08 Jun 2024
Large Language Model-guided Document Selection Xiang Kong Tom Gunter Ruoming Pang 65 4 0 07 Jun 2024
Recovering document annotations for sentence-level bitext R. Wicks Matt Post Philipp Koehn 72 5 0 06 Jun 2024
Enhancing CTC-based speech recognition with diverse modeling units Shiyi Han Zhihong Lei Mingbin Xu Xingyu Na Zhen Huang 83 0 0 05 Jun 2024
StreamSpeech: Simultaneous Speech-to-Speech Translation with Multi-task Learning Shaolei Zhang Qingkai Fang Shoutao Guo Zhengrui Ma Min Zhang Yang Feng 96 10 0 05 Jun 2024
LCS: A Language Converter Strategy for Zero-Shot Neural Machine Translation Zengkui Sun Yijin Liu Fandong Meng Jinan Xu Jinan Xu Jie Zhou 132 2 0 05 Jun 2024
Xmodel-LM Technical Report Yichuan Wang Yang Liu Yu Yan Qun Wang Xucheng Huang Ling Jiang OSLM ALM 48 1 0 05 Jun 2024
Multi-word Term Embeddings Improve Lexical Product Retrieval Viktor Shcherbakov Fedor Krasnov 47 0 0 03 Jun 2024
Applying Intrinsic Debiasing on Downstream Tasks: Challenges and Considerations for Machine Translation Bar Iluz Yanai Elazar Asaf Yehudai Gabriel Stanovsky 65 2 0 02 Jun 2024
An Early Investigation into the Utility of Multimodal Large Language Models in Medical Imaging Sulaiman Khan Md. Rafiul Biswas Alina Murad Hazrat Ali Zubair Shah 91 4 0 02 Jun 2024
$μ$ LO: Compute-Efficient Meta-Generalization of Learned Optimizers Benjamin Thérien Charles-Étienne Joseph Boris Knyazev Edouard Oyallon Irina Rish Eugene Belilovsky AI4CE 131 4 0 31 May 2024
How Multilingual Are Large Language Models Fine-Tuned for Translation? Aquia Richburg Marine Carpuat LRM 78 5 0 30 May 2024
Critical Learning Periods: Leveraging Early Training Dynamics for Efficient Data Pruning E. Chimoto Jay Gala Orevaoghene Ahia Julia Kreutzer Bruce A. Bassett Sara Hooker VLM 95 6 0 29 May 2024
X-VILA: Cross-Modality Alignment for Large Language Model Hanrong Ye De-An Huang Yao Lu Zhiding Yu Ming-Yu Liu ... Jan Kautz Song Han Dan Xu Pavlo Molchanov Hongxu Yin MLLM VLM 86 35 0 29 May 2024
MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series Ge Zhang Scott Qu Jiaheng Liu Chenchen Zhang Chenghua Lin ... Zi-Kai Zhao Jiajun Zhang Wanli Ouyang Wenhao Huang Wenhu Chen ELM 124 46 0 29 May 2024
Integrating Multi-scale Contextualized Information for Byte-based Neural Machine Translation Langlin Huang Yang Feng 77 1 0 29 May 2024
Optimizing Foundation Model Inference on a Many-tiny-core Open-source RISC-V Platform Viviane Potocnik Luca Colagrande Tim Fischer L. Bertaccini Daniele Jahier Pagliari Luca Bompani Luca Benini 87 3 0 29 May 2024
Descriptive Image Quality Assessment in the Wild Zhiyuan You Jinjin Gu Zheyuan Li Xin Cai Kaiwen Zhu Chao Dong Tianfan Xue EGVM 91 22 0 29 May 2024
Wavelet-Based Image Tokenizer for Vision Transformers Zhenhai Zhu Radu Soricut ViT 102 5 0 28 May 2024
Multi-objective Representation for Numbers in Clinical Narratives: A CamemBERT-Bio-Based Alternative to Large-Scale LLMs Boammani Aser Lompo Thanh-Dung Le 101 0 0 28 May 2024
Empowering Character-level Text Infilling by Eliminating Sub-Tokens Houxing Ren Mingjie Zhan Zhongyuan Wu Hongsheng Li AI4CE 72 1 0 27 May 2024
Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization Dixuan Wang Yanda Li Junyuan Jiang Zepeng Ding Ziqin Luo Guochao Jiang Jiaqing Liang Deqing Yang 106 15 0 27 May 2024