Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

29 April 2018

Papers citing "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates"

50 / 628 papers shown

Title
Thunder-Tok: Minimizing Tokens per Word in Tokenizing Korean Texts for Generative Language Models Gyeongje Cho Yeonkyoun So Chanwoo Park Sangmin Lee Sungmok Jung Jaejin Lee VLM 30 0 0 18 Jun 2025
Detecting Hard-Coded Credentials in Software Repositories via LLMs Chidera Biringa Gökhan Kul 31 0 0 16 Jun 2025
One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizers Diana Abagyan Alejandro Salamanca Andres Felipe Cruz-Salinas Kris Cao Hangyu Lin Acyr Locatelli Marzieh Fadaee Ahmet Üstün Sara Hooker CLL 133 0 0 12 Jun 2025
Canonical Autoregressive Generation Ivi Chatzi N. C. Benz Stratis Tsirtsis Manuel Gomez Rodriguez 15 0 0 06 Jun 2025
ViCocktail: Automated Multi-Modal Data Collection for Vietnamese Audio-Visual Speech Recognition Thai-Binh Nguyen T. Nguyen Quoc Truong Do Chi Mai Luong 86 0 0 05 Jun 2025
TokAlign: Efficient Vocabulary Adaptation via Token Alignment Chong Li Jiajun Zhang Chengqing Zong VLM 59 0 0 04 Jun 2025
Beyond Text Compression: Evaluating Tokenizers Across Scales Jonas F. Lotz António V. Lopes Stephan Peitz Hendra Setiawan Leonardo Emili 61 0 0 03 Jun 2025
Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics Ella Rannon David Burstein AI4TS 26 0 0 02 Jun 2025
StochasTok: Improving Fine-Grained Subword Understanding in LLMs Anya Sims Thom Foster Klara Kaleb Tuan-Duy H. Nguyen Joseph Lee Jakob N. Foerster Yee Whye Teh Cong Lu 92 1 0 02 Jun 2025
The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence Marco Gaido Sara Papi L. Bentivogli Alessio Brutti Mauro Cettolo R. Gretter M. Matassoni Mohamed Nabih Matteo Negri 47 0 0 29 May 2025
Spontaneous Speech Variables for Evaluating LLMs Cognitive Plausibility Sheng-Fu Wang Laurent Prevot Jou-an Chi Ri-Sheng Huang Shu-Kai Hsieh LRM 67 0 0 22 May 2025
BanglaByT5: Byte-Level Modelling for Bangla Pramit Bhattacharyya Arnab Bhattacharya 47 0 0 21 May 2025
GIF: Generative Inspiration for Face Recognition at Scale Saeed Ebrahimi Sahar Rahimi Ali Dabouei Srinjoy Das Jeremy M. Dawson Nasser M. Nasrabadi CVBM 554 0 0 05 May 2025
An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding Xiuwei Shang Zhenkan Fu Shaoyin Cheng Guoqiang Chen Gangyang Li Li Hu Weinan Zhang N. Yu 98 0 0 30 Apr 2025
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization Enes Özeren Yihong Liu Hinrich Schütze 69 0 0 21 Apr 2025
TASTE: Text-Aligned Speech Tokenization and Embedding for Spoken Language Modeling Liang-Hsuan Tseng Yi-Chang Chen Kuan-Yi Lee Da-shan Shiu Hung-yi Lee AuLLM 161 0 0 09 Apr 2025
Limitations of Religious Data and the Importance of the Target Domain: Towards Machine Translation for Guinea-Bissau Creole Jacqueline Rowe Edward Gow-Smith Mark Hepple 112 0 0 03 Apr 2025
Overcoming Vocabulary Constraints with Pixel-level Fallback Jonas F. Lotz Hendra Setiawan Stephan Peitz Yova Kementchedjhieva 100 1 0 02 Apr 2025
From Smør-re-brød to Subwords: Training LLMs on Danish, One Morpheme at a Time Mikkel Wildner Kildeberg Emil Allerslev Schledermann Nicolaj Larsen Rob van der Goot 78 0 0 02 Apr 2025
Tokenization of Gaze Data Tim Rolff Jurik Karimian Niklas Hypki S. Schmidt Markus Lappe Frank Steinicke 118 0 0 28 Mar 2025
ParsiPy: NLP Toolkit for Historical Persian Texts in Python Farhan Farsi Parnian Fazel Sepand Haghighi Sadra Sabouri Farzaneh Goshtasb Nadia Hajipour Ehsaneddin Asgari Hossein Sameti 68 0 0 22 Mar 2025
Splintering Nonconcatenative Languages for Better Tokenization Bar Gazit Shaltiel Shmidman Avi Shmidman Yuval Pinter 104 0 0 18 Mar 2025
Annotating and Inferring Compositional Structures in Numeral Systems Across Languages Arne Rubehn Christoph Rzymski Luca Ciucci Kellen Parker van Dam Alžběta Kučerová Katja Bocklage David Snee Abishek Stephen Johann-Mattis List 73 0 0 03 Mar 2025
Scaling Laws for Downstream Task Performance in Machine Translation Berivan Isik Natalia Ponomareva Hussein Hazimeh Dimitris Paparas Sergei Vassilvitskii Sanmi Koyejo 165 23 0 24 Feb 2025
Deterministic Reversible Data Augmentation for Neural Machine Translation Jiashu Yao Heyan Huang Zeming Liu Yuhang Guo 172 0 0 21 Feb 2025
PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference Burc Gokden 118 0 0 19 Feb 2025
TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking Shahriar Kabir Nahin R. N. Nandi Sagor Sarker Quazi Sarwar Muhtaseem Md. Kowsher Apu Chandraw Shill Md Ibrahim Mehadi Hasan Menon Tareq Al Muntasir Firoj Alam 189 0 0 16 Feb 2025
Enhancing LLM Character-Level Manipulation via Divide and Conquer Zhen Xiong Yujun Cai Bryan Hooi Nanyun Peng Kai-Wei Chang Zhecheng Li 162 0 0 12 Feb 2025
Beyond Literal Token Overlap: Token Alignability for Multilinguality Katharina Hämmerl Tomasz Limisiewicz Jindrich Libovický Alexander Fraser 73 0 0 10 Feb 2025
Scaling Embedding Layers in Language Models Da Yu Edith Cohen Badih Ghazi Yangsibo Huang Pritish Kamath Ravi Kumar Daogao Liu Chiyuan Zhang 172 1 0 03 Feb 2025
Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures Gabriel Lindenmaier Sean Papay Sebastian Padó 154 0 0 02 Feb 2025
Accelerating LLM Inference with Lossless Speculative Decoding Algorithms for Heterogeneous Vocabularies Nadav Timor Jonathan Mamou Daniel Korat Moshe Berchansky Oren Pereg Gaurav Jain Roy Schwartz Moshe Wasserblat 189 3 0 31 Jan 2025
Weight-based Analysis of Detokenization in Language Models: Understanding the First Stage of Inference Without Inference Go Kamoda Benjamin Heinzerling Tatsuro Inaba Keito Kudo Keisuke Sakaguchi Kentaro Inui MILM 133 3 0 27 Jan 2025
Multi-Objective Hyperparameter Selection via Hypothesis Testing on Reliability Graphs Amirmohammad Farzaneh Osvaldo Simeone 520 0 0 22 Jan 2025
Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directions Doaa Mahmud Hadeel Hajmohamed Shamma Almentheri Shamma Alqaydi Lameya Aldhaheri R. A. Khalil Nasir Saeed AI4TS 99 12 0 08 Jan 2025
LLM-jp: A Cross-organizational Project for the Research and Development of Fully Open Japanese LLMs LLM-jp Akiko Aizawa Eiji Aramaki Bowen Chen Fei Cheng ... Yuya Yamamoto Yusuke Yamauchi Hitomi Yanaka Rio Yokota Koichiro Yoshino 111 17 0 31 Dec 2024
Mask Factory: Towards High-quality Synthetic Data Generation for Dichotomous Image Segmentation Haotian Qian YD Chen Shengtao Lou Fahad Shahbaz Khan Xiaogang Jin Deng-Ping Fan DiffM 83 5 0 26 Dec 2024
The Language of Motion: Unifying Verbal and Non-verbal Language of 3D Human Motion Changan Chen Juze Zhang S. K. Lakshmikanth Yusu Fang Ruizhi Shao Gordon Wetzstein L. Fei-Fei Ehsan Adeli VGen 135 5 0 13 Dec 2024
Efficient Continual Pre-training of LLMs for Low-resource Languages Arijit Nag Soumen Chakrabarti Animesh Mukherjee Niloy Ganguly 117 1 0 13 Dec 2024
Multi-Head Encoding for Extreme Label Classification Daojun Liang Haixia Zhang Dongfeng Yuan Minggao Zhang 118 0 0 13 Dec 2024
From Language Models over Tokens to Language Models over Characters Tim Vieira Ben LeBrun Mario Giulianelli Juan Luis Gastaldi Brian DuSell John Terilla Timothy J. O'Donnell Ryan Cotterell 156 13 0 04 Dec 2024
Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods Burak Suyunu Enes Taylan Arzucan Özgür 99 3 0 26 Nov 2024
Xmodel-1.5: An 1B-scale Multilingual LLM Wang Qun Liu Yang Lin Qingquan Jiang Ling LRM 71 0 0 15 Nov 2024
Number Cookbook: Number Understanding of Language Models and How to Improve It Haotong Yang Yi Hu Shijia Kang Zhouchen Lin Muhan Zhang LRM 112 8 0 06 Nov 2024
MoCE: Adaptive Mixture of Contextualization Experts for Byte-based Neural Machine Translation Langlin Huang Mengyu Bu Yang Feng 101 0 0 03 Nov 2024
Morphological Typology in BPE Subword Productivity and Language Modeling Iñigo Parra 61 0 0 31 Oct 2024
PLDR-LLM: Large Language Model from Power Law Decoder Representations Burc Gokden 59 1 0 22 Oct 2024
Neural Search Space in Gboard Decoder Yanxiang Zhang Yuanbo Zhang Haicheng Sun Yun Wang Billy Dou Gary Sivek Shumin Zhai 102 0 0 21 Oct 2024
LLM The Genius Paradox: A Linguistic and Math Expert's Struggle with Simple Word-based Counting Problems Nan Xu Xuezhe Ma LRM 160 0 0 18 Oct 2024
Qtok: A Comprehensive Framework for Evaluating Multilingual Tokenizer Quality in Large Language Models Iaroslav Chelombitko Egor Safronov Aleksey Komissarov 76 1 0 16 Oct 2024