Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates

29 April 2018

Papers citing "Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates"

50 / 628 papers shown

Title
Zero-Shot Tokenizer Transfer Benjamin Minixhofer Edoardo Ponti Ivan Vulić VLM 83 13 0 13 May 2024
Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models Sander Land Max Bartolo 116 25 0 08 May 2024
Revisiting N-Gram Models: Their Impact in Modern Neural Networks for Handwritten Text Recognition Solène Tarride Christopher Kermorvant 72 1 0 30 Apr 2024
A cost minimization approach to fix the vocabulary size in a tokenizer for an End-to-End ASR system Sunil Kumar Kopparapu Ashish Panda 53 0 0 29 Apr 2024
Can Perplexity Predict Fine-tuning Performance? An Investigation of Tokenization Effects on Sequential Language Models for Nepali Nishant Luitel Nirajan Bekoju Anand Kumar Sah Subarna Shakya 112 1 0 28 Apr 2024
Act as a Honeytoken Generator! An Investigation into Honeytoken Generation with Large Language Models Daniel Reti Norman Becker Tillmann Angeli Anasuya Chattopadhyay Daniel Schneider Sebastian Vollmer Hans D. Schotten 77 5 0 24 Apr 2024
Evaluating Subword Tokenization: Alien Subword Composition and OOV Generalization Challenge Khuyagbaatar Batsuren Ekaterina Vylomova Verna Dankers Tsetsuukhei Delgerbaatar Omri Uzan Yuval Pinter Gábor Bella 96 10 0 20 Apr 2024
Simultaneous Interpretation Corpus Construction by Large Language Models in Distant Language Pair Yusuke Sakai Mana Makinae Hidetaka Kamigaito Taro Watanabe 103 5 0 18 Apr 2024
On the Effect of (Near) Duplicate Subwords in Language Modelling Anton Schäfer Thomas Hofmann Imanol Schlag Tiago Pimentel 70 2 0 09 Apr 2024
Training LLMs over Neurally Compressed Text Brian Lester Jaehoon Lee A. Alemi Jeffrey Pennington Adam Roberts Jascha Narain Sohl-Dickstein Noah Constant 92 7 0 04 Apr 2024
Dynamic Neural Control Flow Execution: An Agent-Based Deep Equilibrium Approach for Binary Vulnerability Detection Litao Li Steven H. H. Ding Andrew Walenstein P. Charland Benjamin C. M. Fung 58 0 0 03 Apr 2024
Revisiting subword tokenization: A case study on affixal negation in large language models Thinh Hung Truong Yulia Otmakhova Karin Verspoor Trevor Cohn Timothy Baldwin 77 2 0 03 Apr 2024
Forklift: An Extensible Neural Lifter Jordi Armengol-Estapé Rodrigo C. O. Rocha Jackson Woodruff Pasquale Minervini Michael F. P. O'Boyle 63 1 0 01 Apr 2024
An Analysis of BPE Vocabulary Trimming in Neural Machine Translation Marco Cognetta Tatsuya Hiraoka Naoaki Okazaki Rico Sennrich Yuval Pinter 109 2 0 30 Mar 2024
A Systematic Analysis of Subwords and Cross-Lingual Transfer in Multilingual Translation Francois Meyer Jan Buys 90 1 0 29 Mar 2024
AlloyBERT: Alloy Property Prediction with Large Language Models Akshat Chaudhari Chakradhar Guntuboina Hongshuo Huang A. Farimani 119 4 0 28 Mar 2024
Homogeneous Tokenizer Matters: Homogeneous Visual Tokenizer for Remote Sensing Image Understanding Run Shao Zhaoyang Zhang Chao Tao Yunsheng Zhang Chengli Peng Haifeng Li VLM 81 6 0 27 Mar 2024
Can Language Beat Numerical Regression? Language-Based Multimodal Trajectory Prediction Inhwan Bae Junoh Lee Hae-Gon Jeon 105 22 0 27 Mar 2024
Provably Secure Disambiguating Neural Linguistic Steganography Yuang Qi Kejiang Chen Kai Zeng Weiming Zhang Neng H. Yu 43 5 0 26 Mar 2024
Cross-lingual Contextualized Phrase Retrieval Huayang Li Deng Cai Zhi Qu Qu Cui Hidetaka Kamigaito Lemao Liu Taro Watanabe 56 0 0 25 Mar 2024
Synthetic Data Generation and Joint Learning for Robust Code-Mixed Translation Kamal Kumar Yinhan Liu Parth Patwa Tanmoy Mihir Adam Roberts 95 2 0 25 Mar 2024
More than Just Statistical Recurrence: Human and Machine Unsupervised Learning of Māori Word Segmentation across Morphological Processes A. Varatharaj Simon Todd 32 0 0 21 Mar 2024
Exploring Tokenization Strategies and Vocabulary Sizes for Enhanced Arabic Language Models M. Alrefaie Nour Eldin Morsy Nada Samir 59 6 0 17 Mar 2024
Using Contextual Information for Sentence-level Morpheme Segmentation Prabin Bhandari Abhishek Paudel 28 1 0 15 Mar 2024
Token Alignment via Character Matching for Subword Completion Ben Athiwaratkun Shiqi Wang Mingyue Shang Yuchen Tian Zijian Wang Sujan Kumar Gonugondla Sanjay Krishna Gouda Rob Kwiatowski Ramesh Nallapati Bing Xiang 99 6 0 13 Mar 2024
Triples-to-isiXhosa (T2X): Addressing the Challenges of Low-Resource Agglutinative Data-to-Text Generation Francois Meyer Jan Buys 57 2 0 12 Mar 2024
MAMMOTH: Massively Multilingual Modular Open Translation @ Helsinki Timothee Mickus Stig-Arne Gronroos Joseph Attieh M. Boggia Ona de Gibert Shaoxiong Ji Niki Andreas Lopi Alessandro Raganato Raúl Vázquez Jörg Tiedemann 57 4 0 12 Mar 2024
Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance Omer Goldman Avi Caciularu Matan Eyal Kris Cao Idan Szpektor Reut Tsarfaty 115 31 0 10 Mar 2024
Authorship Attribution in Bangla Literature (AABL) via Transfer Learning using ULMFiT Aisha Khatun Anisur Rahman Md. Saiful Islam Hemayet Ahmed Chowdhury A. Tasnim 69 3 0 08 Mar 2024
Did Translation Models Get More Robust Without Anyone Even Noticing? Ben Peters André F. T. Martins 66 3 0 06 Mar 2024
A Generative Approach for Wikipedia-Scale Visual Entity Recognition Mathilde Caron Ahmet Iscen Alireza Fathi Cordelia Schmid 94 5 0 04 Mar 2024
Transformers for Low-Resource Languages:Is Féidir Linn! Séamus Lankford H. Alfi Tamás Sarlós 93 17 0 04 Mar 2024
Language and Speech Technology for Central Kurdish Varieties Sina Ahmadi Daban Q. Jaff Md Mahfuz Ibn Alam Antonios Anastasopoulos 99 2 0 04 Mar 2024
adaptNMT: an open-source, language-agnostic development environment for Neural Machine Translation Séamus Lankford Haithem Afli Andy Way 73 3 0 04 Mar 2024
Human Evaluation of English--Irish Transformer-Based NMT Séamus Lankford Haithem Afli Andy Way 124 11 0 04 Mar 2024
VBART: The Turkish LLM Meliksah Turker Mehmet Erdi Ari Aydin Han VLM 44 4 0 02 Mar 2024
Greed is All You Need: An Evaluation of Tokenizer Inference Methods Omri Uzan Craig W. Schmidt Chris Tanner Yuval Pinter 79 17 0 02 Mar 2024
Rethinking Tokenization: Crafting Better Tokenizers for Large Language Models Jinbiao Yang LLMAG 169 11 0 01 Mar 2024
Heavy-Tailed Class Imbalance and Why Adam Outperforms Gradient Descent on Language Models Frederik Kunstner Robin Yadav Alan Milligan Mark Schmidt Alberto Bietti 95 34 0 29 Feb 2024
Beyond Language Models: Byte Models are Digital World Simulators Shangda Wu Xu Tan Zili Wang Rui Wang Xiaobing Li Maosong Sun 65 13 0 29 Feb 2024
CEBin: A Cost-Effective Framework for Large-Scale Binary Code Similarity Detection Hao Wang Zeyu Gao Chao Zhang Mingyang Sun Yuchen Zhou Han Qiu Xiangwei Xiao 65 13 0 29 Feb 2024
Tokenization Is More Than Compression Craig W. Schmidt Varshini Reddy Haoran Zhang Alec Alameddine Omri Uzan Yuval Pinter Chris Tanner 124 38 0 28 Feb 2024
Natural Language Processing Methods for Symbolic Music Generation and Information Retrieval: a Survey Dinh-Viet-Toan Le Louis Bigo Mikaela Keller Dorien Herremans MedIm 91 14 0 27 Feb 2024
CLAP: Learning Transferable Binary Code Representations with Natural Language Supervision Hao Wang Zeyu Gao Chao Zhang Zihan Sha Mingyang Sun Yuchen Zhou Wenyu Zhu Wenju Sun Han Qiu Xiangwei Xiao 92 22 0 26 Feb 2024
How Important Is Tokenization in French Medical Masked Language Models? Yanis Labrak Adrien Bazoge B. Daille Mickael Rouvier Richard Dufour 66 1 0 22 Feb 2024
Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs Aaditya K. Singh DJ Strouse 116 61 0 22 Feb 2024
The Impact of Word Splitting on the Semantic Content of Contextualized Word Representations Aina Garí Soler Matthieu Labeau Chloé Clavel VLM 75 2 0 22 Feb 2024
Two Counterexamples to Tokenization and the Noiseless Channel Marco Cognetta Vilém Zouhar Sangwhan Moon Naoaki Okazaki 51 0 0 22 Feb 2024
Subobject-level Image Tokenization Delong Chen Samuel Cahyawijaya Jianfeng Liu Baoyuan Wang Pascale Fung VLM OCL 278 9 0 22 Feb 2024
Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions? Alexander Arno Weber Klaudia Thellmann Jan Ebert Nicolas Flores-Herr Jens Lehmann Michael Fromm Mehdi Ali 72 5 0 21 Feb 2024