SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

19 August 2018

Papers citing "SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing"

50 / 1,950 papers shown

Title
SocialGen: Modeling Multi-Human Social Interaction with Language Models Heng Yu Juze Zhang Changan Chen Tiange Xiang Yusu Fang Juan Carlos Niebles Ehsan Adeli VGen 93 1 0 28 Mar 2025
Tokenization of Gaze Data Tim Rolff Jurik Karimian Niklas Hypki S. Schmidt Markus Lappe Frank Steinicke 109 0 0 28 Mar 2025
Beyond Words: Advancing Long-Text Image Generation via Multimodal Autoregressive Models Alex Jinpeng Wang Linjie Li Zhiyong Yang Lijuan Wang Min Li DiffM 103 0 0 26 Mar 2025
Named Entity Recognition in Context Colin Brisson Ayoub Kahfy Marc Bui Frédéric Constant 138 0 0 26 Mar 2025
Gemma 3 Technical Report Gemma Team Aishwarya B Kamath Johan Ferret Shreya Pathak Nino Vieillard ... Harshal Tushar Lehri Hussein Hazimeh Ian Ballantyne Idan Szpektor Ivan Nardini VLM 193 136 0 25 Mar 2025
Payload-Aware Intrusion Detection with CMAE and Large Language Models Yongcheol Kim Chanjae Lee Young Yoon 77 0 0 23 Mar 2025
KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications M. Bommarito Daniel Martin Katz Jillian Bommarito 82 1 0 21 Mar 2025
Self-Vocabularizing Training for Neural Machine Translation Pin-Jie Lin Ernie Chang Yangyang Shi Vikas Chandra 128 0 0 18 Mar 2025
SuperBPE: Space Travel for Language Models Alisa Liu J. Hayase Valentin Hofmann Sewoong Oh Noah A. Smith Yejin Choi 157 10 0 17 Mar 2025
Unified Autoregressive Visual Generation and Understanding with Continuous Tokens Lijie Fan Luming Tang Siyang Qin Tianhong Li Xuan S. Yang ... Tao Zhu Michael Rubinstein Michalis Raptis Deqing Sun Radu Soricut 125 8 0 17 Mar 2025
Plausibility Vaccine: Injecting LLM Knowledge for Event Plausibility Jacob Chmura Jonah Dauvet Sebastian Sabry 88 0 0 16 Mar 2025
Florenz: Scaling Laws for Systematic Generalization in Vision-Language Models Julian Spravil Sebastian Houben Sven Behnke VLM 197 0 0 12 Mar 2025
BPQA Dataset: Evaluating How Well Language Models Leverage Blood Pressures to Answer Biomedical Questions Chi Hang Ruiqi Deng L. Jiang Zihao Yang Anton Alyakin Daniel Alber E. Oermann AI4MH LM&MA 89 0 0 06 Mar 2025
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models Catherine Arnett Tyler A. Chang J. Michaelov Benjamin Bergen 90 0 0 05 Mar 2025
LLM-Safety Evaluations Lack Robustness Tim Beyer Sophie Xhonneux Simon Geisler Gauthier Gidel Leo Schwinn Stephan Günnemann ALM ELM 485 2 0 04 Mar 2025
SkipPipe: Partial and Reordered Pipelining Framework for Training LLMs in Heterogeneous Networks Nikolay Blagoev Lydia Yiyu Chen Oğuzhan Ersoy 99 2 0 27 Feb 2025
A City of Millions: Mapping Literary Social Networks At Scale Sil Hamilton Rebecca M. M. Hicke David M. Mimno Matthew Wilkens GNN 466 1 0 26 Feb 2025
(Mis)Fitting: A Survey of Scaling Laws Margaret Li Sneha Kudugunta Luke Zettlemoyer 138 4 0 26 Feb 2025
Lost in Space: Optimizing Tokens for Grammar-Constrained Decoding Sil Hamilton David Mimno 104 0 0 24 Feb 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 163 23 0 24 Feb 2025
Deterministic Reversible Data Augmentation for Neural Machine Translation Jiashu Yao Heyan Huang Zeming Liu Yuhang Guo 172 0 0 21 Feb 2025
Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models Lior Belenki Alekh Agarwal Tianze Shi Kristina Toutanova MoE 97 0 0 21 Feb 2025
PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference Burc Gokden 118 0 0 19 Feb 2025
From Principles to Applications: A Comprehensive Survey of Discrete Tokenizers in Generation, Comprehension, Recommendation, and Information Retrieval Jian Jia Jingtong Gao Ben Xue Junhao Wang Qingpeng Cai Quan Chen Xiangyu Zhao Peng Jiang Kun Gai OffRL 145 2 0 18 Feb 2025
Baichuan-M1: Pushing the Medical Capability of Large Language Models Binghai Wang Haizhou Zhao Huozhi Zhou Liang Song Mingyu Xu ... Yan Zhang Yifei Duan Yuyan Zhou Zhi-Ming Ma Zhikai Wu LM&MA ELM AI4MH 121 10 0 18 Feb 2025
TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking Shahriar Kabir Nahin R. N. Nandi Sagor Sarker Quazi Sarwar Muhtaseem Md. Kowsher Apu Chandraw Shill Md Ibrahim Mehadi Hasan Menon Tareq Al Muntasir Firoj Alam 185 0 0 16 Feb 2025
Enhancing LLM Character-Level Manipulation via Divide and Conquer Zhen Xiong Yujun Cai Bryan Hooi Nanyun Peng Kai-Wei Chang Zhecheng Li 162 0 0 12 Feb 2025
A Differentiable Alignment Framework for Sequence-to-Sequence Modeling via Optimal Transport Yacouba Kaloga Shashi Kumar P. Motlícek Ina Kodrasi OT 102 0 0 03 Feb 2025
Scaling Embedding Layers in Language Models Da Yu Edith Cohen Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Daogao Liu Chiyuan Zhang 168 1 0 03 Feb 2025
Vision-centric Token Compression in Large Language Model Ling Xing Alex Jinpeng Wang Rui Yan Xiangbo Shu Jinhui Tang VLM 155 0 0 02 Feb 2025
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies Nadav Timor Jonathan Mamou Daniel Korat Moshe Berchansky Oren Pereg Gaurav Jain Roy Schwartz Moshe Wasserblat 189 3 0 31 Jan 2025
BLR-MoE: Boosted Language-Routing Mixture of Experts for Domain-Robust Multilingual E2E ASR Guodong Ma Wenxuan Wang Lifeng Zhou Yuting Yang Yuke Li Binbin Du MoE 129 0 0 22 Jan 2025
aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing Siyuan Jiang Jia Li He Zong Huanyu Liu Hao Zhu ... Wei Ning G. Wang Yihong Dong Kechi Zhang Ge Li ALM 98 2 0 17 Jan 2025
BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs Sheng Zhang Yanbo Xu Naoto Usuyama Hanwen Xu J. Bagga ... Carlo Bifulco M. Lungren Tristan Naumann Sheng Wang Hoifung Poon LM&MA MedIm 240 235 0 10 Jan 2025
Registering Source Tokens to Target Language Spaces in Multilingual Neural Machine Translation Zhi Qu Yiran Wang Jiannan Mao Chenchen Ding Hideki Tanaka Masao Utiyama Taro Watanabe LRM 124 0 0 06 Jan 2025
On LLM-Enhanced Mixed-Type Data Imputation with High-Order Message Passing Jinqiao Wang Kai Wang Yanzhe Zhang Wenjie Zhang Xiwei Xu Xuemin Lin 115 4 0 04 Jan 2025
Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison Tsz Kin Lam Marco Gaido Sara Papi L. Bentivogli Barry Haddow 128 0 0 04 Jan 2025
A Modular-based Strategy for Mitigating Gradient Conflicts in Simultaneous Speech Translation Xiaoqian Liu Yangfan Du Jiadong Wang Yuan Ge Chen Xu Tong Xiao Guocheng Chen Jingbo Zhu 139 0 0 31 Dec 2024
ImagePiece: Content-aware Re-tokenization for Efficient Image Recognition Seungdong Yoa Seungjun Lee Hyeseung Cho Bumsoo Kim Woohyung Lim ViT 98 0 0 21 Dec 2024
ECG-Byte: A Tokenizer for End-to-End Generative Electrocardiogram Language Modeling William Jongwon Han Chaojing Duan M. Rosenberg Emerson Liu Ding Zhao 138 1 0 18 Dec 2024
Extending LLMs to New Languages: A Case Study of Llama and Persian Adaptation Samin Mahdizadeh Sani Pouya Sadeghi Thuy-Trang Vu Yadollah Yaghoobzadeh Gholamreza Haffari 181 2 0 17 Dec 2024
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion Changan Chen Juze Zhang S. K. Lakshmikanth Yusu Fang Ruizhi Shao Gordon Wetzstein L. Fei-Fei Ehsan Adeli VGen 133 5 0 13 Dec 2024
Efficient Continual Pre-training of LLMs for Low-resource Languages Arijit Nag Soumen Chakrabarti Animesh Mukherjee Niloy Ganguly 115 1 0 13 Dec 2024
Multi-Head Encoding for Extreme Label Classification Daojun Liang Haixia Zhang Dongfeng Yuan Minggao Zhang 113 0 0 13 Dec 2024
PolyIPA -- Multilingual Phoneme-to-Grapheme Conversion Model Davor Lauc 106 0 0 12 Dec 2024
Scaling Sequential Recommendation Models with Transformers Pablo Zivic Hernán Ceferino Vázquez Jorge Sanchez OffRL LRM 114 1 0 10 Dec 2024
Representation Purification for End-to-End Speech Translation Chengwei Zhang Yue Zhou Rui Zhao Yidong Chen Xiaodong Shi 78 0 0 05 Dec 2024
From Language Models over Tokens to Language Models over Characters Tim Vieira Ben LeBrun Mario Giulianelli Juan Luis Gastaldi Brian DuSell John Terilla Timothy J. O'Donnell Ryan Cotterell 156 13 0 04 Dec 2024
Yi-Lightning Technical Report 01. AI : Alan Wake Albert Wang Bei Chen ... Yuxuan Sha Zhaodong Yan Zhiyuan Liu Zirui Zhang Zonghong Dai OSLM 209 4 0 02 Dec 2024
A Wave is Worth 100 Words: Investigating Cross-Domain Transferability in Time Series Xiangkai Ma Xiaobin Hong Wenzhong Li Sanglu Lu AI4TS 138 0 0 01 Dec 2024