v1v2 (latest)

Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models

24 May 2022

Luke Zettlemoyer

Papers citing "Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models"

22 / 22 papers shown

Title
Domain Pre-training Impact on Representations César González-Gutiérrez A. Quattoni 38 0 0 30 May 2025
Semantic Pivots Enable Cross-Lingual Transfer in Large Language Models Kaiyu He Tong Zhou Yubo Chen Delai Qiu Shengping Liu Kang Liu Jun Zhao LRM 73 0 0 22 May 2025
Tracing Multilingual Factual Knowledge Acquisition in Pretraining Yihong Liu Mingyang Wang Amir Hossein Kargaran Felicia Körner Ercong Nie Yun Xue François Yvon Hinrich Schutze HILM KELM 130 0 0 20 May 2025
On the Acquisition of Shared Grammatical Representations in Bilingual Language Models Catherine Arnett Tyler A. Chang J. Michaelov Benjamin Bergen 90 0 0 05 Mar 2025
Deploying Multi-task Online Server with Large Language Model Yincen Qu Chao Ma Xiangying Dai Hui Zhou Yiting Wu Hengyue Liu 58 0 0 06 Nov 2024
Modular Sentence Encoders: Separating Language Specialization from Cross-Lingual Alignment Yongxin Huang Kexin Wang Goran Glavaš Iryna Gurevych 100 1 0 20 Jul 2024
Probing the Emergence of Cross-lingual Alignment during LLM Training Hetong Wang Pasquale Minervini Edoardo Ponti 139 15 0 19 Jun 2024
Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations Ziqiao Ma Zekun Wang Joyce Chai 146 4 0 22 May 2024
Multi-Task Contrastive Learning for 8192-Token Bilingual Text Embeddings Isabelle Mohr Markus Krimmel Saba Sturua Mohammad Kalim Akram Andreas Koukounas ... Susana Guzman Bo Wang Maximilian Werk Nan Wang Han Xiao 84 16 0 26 Feb 2024
Breaking the Curse of Multilinguality with Cross-lingual Expert Language Models Terra Blevins Tomasz Limisiewicz Suchin Gururangan Margaret Li Hila Gonen Noah A. Smith Luke Zettlemoyer 100 27 0 19 Jan 2024
When Is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages Tyler A. Chang Catherine Arnett Zhuowen Tu Benjamin Bergen LRM 142 8 0 15 Nov 2023
OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretraining Yihong Liu Peiqin Lin Mingyang Wang Hinrich Schütze 71 29 0 15 Nov 2023
Counterfactually Probing Language Identity in Multilingual Models Anirudh Srinivasan Venkata S Govindarajan Kyle Mahowald 70 1 0 29 Oct 2023
Unveiling Multilinguality in Transformer Models: Exploring Language Specificity in Feed-Forward Networks Sunit Bhattacharya Ondrej Bojar 49 12 0 24 Oct 2023
Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability Tyler A. Chang Zhuowen Tu Benjamin Bergen 59 13 0 29 Aug 2023
Second Language Acquisition of Neural Language Models Miyu Oba Tatsuki Kuribayashi Hiroki Ouchi Taro Watanabe 63 6 0 05 Jun 2023
How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning Rochelle Choenni Dan Garrette Ekaterina Shutova 100 16 0 22 May 2023
Training Trajectories of Language Models Across Scales Mengzhou Xia Mikel Artetxe Chunting Zhou Xi Lin Ramakanth Pasunuru Danqi Chen Luke Zettlemoyer Ves Stoyanov AIFin LRM 98 64 0 19 Dec 2022
Data-Efficient Cross-Lingual Transfer with Language-Specific Subnetworks Rochelle Choenni Dan Garrette Ekaterina Shutova 94 2 0 31 Oct 2022
WeLM: A Well-Read Pre-trained Language Model for Chinese Hui Su Xiao Zhou Houjin Yu Xiaoyu Shen Yuwen Chen Zilin Zhu Yang Yu Jie Zhou 87 23 0 21 Sep 2022
Language Modelling with Pixels Phillip Rust Jonas F. Lotz Emanuele Bugliarello Elizabeth Salesky Miryam de Lhoneux Desmond Elliott VLM 107 46 0 14 Jul 2022
Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models Kushal Tirumala Aram H. Markosyan Luke Zettlemoyer Armen Aghajanyan TDI 127 197 0 22 May 2022