An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference

16 February 2024

Papers citing "An Empirical Study on Cross-lingual Vocabulary Adaptation for Efficient Language Model Inference"

6 / 6 papers shown

Title
Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning Shaurya Sharthak Vinayak Pahalwan Adithya Kamath Adarsh Shirawalmath CLL VLM 45 0 0 14 May 2025
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization Enes Özeren Yihong Liu Hinrich Schütze 31 0 0 21 Apr 2025
Efficient Continual Pre-training of LLMs for Low-resource Languages Arijit Nag Soumen Chakrabarti Animesh Mukherjee Niloy Ganguly 82 0 0 13 Dec 2024
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? HyoJung Han Akiko Eriguchi Haoran Xu Hieu T. Hoang Marine Carpuat Huda Khayrallah VLM 37 2 0 12 Oct 2024
How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models Phillip Rust Jonas Pfeiffer Ivan Vulić Sebastian Ruder Iryna Gurevych 80 235 0 31 Dec 2020
When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Models Benjamin Muller Antonis Anastasopoulos Benoît Sagot Djamé Seddah LRM 134 165 0 24 Oct 2020