OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient
Large-scale Multilingual Continued Pretraining

OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining

15 November 2023

Hinrich Schütze

Papers citing "OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining"

11 / 11 papers shown

Title
Bielik v3 Small: Technical Report Krzysztof Ociepa Łukasz Flis Remigiusz Kinas Krzysztof Wróbel Adrian Gwoździej 27 0 0 05 May 2025
HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization Enes Özeren Yihong Liu Hinrich Schütze 31 0 0 21 Apr 2025
Adapters for Altering LLM Vocabularies: What Languages Benefit the Most? HyoJung Han Akiko Eriguchi Haoran Xu Hieu T. Hoang Marine Carpuat Huda Khayrallah VLM 37 2 0 12 Oct 2024
An Empirical Comparison of Vocabulary Expansion and Initialization Approaches for Language Models Nandini Mundra Aditya Nanda Kishore Raj Dabre Ratish Puduppully Anoop Kunchukuttan Mitesh Khapra 30 3 0 08 Jul 2024
A Recipe of Parallel Corpora Exploitation for Multilingual Large Language Models Peiqin Lin André F. T. Martins Hinrich Schütze 61 2 0 29 Jun 2024
XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples Peiqin Lin André F. T. Martins Hinrich Schütze RALM 50 2 0 08 May 2024
ANGOFA: Leveraging OFA Embedding Initialization and Synthetic Data for Angolan Language Model Osvaldo Luamba Quinjica David Ifeoluwa Adelani 35 0 0 03 Apr 2024
A study of conceptual language similarity: comparison and evaluation Haotian Ye Yihong Liu Hinrich Schütze 32 2 0 22 May 2023
Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models Terra Blevins Hila Gonen Luke Zettlemoyer LRM 59 26 0 24 May 2022
Rethinking embedding coupling in pre-trained language models Hyung Won Chung Thibault Févry Henry Tsai Melvin Johnson Sebastian Ruder 95 142 0 24 Oct 2020
Multilingual BERT Post-Pretraining Alignment Lin Pan Chung-Wei Hang Haode Qi Abhishek Shah Saloni Potdar Mo Yu 114 44 0 23 Oct 2020