Transfer of Structural Knowledge from Synthetic Languages

This work explores transfer learning from several synthetic languages to English. We investigate the structure of the embeddings in the fine-tuned models, the information they contain, and the capabilities of the fine-tuned models on simple linguistic tasks. We also introduce a new synthetic language that leads to better transfer to English than the languages used in previous research. Finally, we introduce Tiny-Cloze Benchmark - a new synthetic benchmark for natural language understanding that is more informative for less powerful models. We use Tiny-Cloze Benchmark to evaluate fine-tuned models in several domains demonstrating that fine-tuning on a new synthetic language allows for better performance on a variety of tasks.

@article{budnikov2025_2505.15769,
  title={ Transfer of Structural Knowledge from Synthetic Languages },
  author={ Mikhail Budnikov and Ivan Yamshchikov },
  journal={arXiv preprint arXiv:2505.15769},
  year={ 2025 }
}