31
0

Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation

Main:8 Pages
12 Figures
Bibliography:2 Pages
11 Tables
Appendix:3 Pages
Abstract

The success of building textless speech-to-speech translation (S2ST) models has attracted much attention. However, S2ST still faces two main challenges: 1) extracting linguistic features for various speech signals, called cross-modal (CM), and 2) learning alignment of difference languages in long sequences, called cross-lingual (CL). We propose the unit language to overcome the two modeling challenges. The unit language can be considered a text-like representation format, constructed using nn-gram language modeling. We implement multi-task learning to utilize the unit language in guiding the speech modeling process. Our initial results reveal a conflict when applying source and target unit languages simultaneously. We propose task prompt modeling to mitigate this conflict. We conduct experiments on four languages of the Voxpupil dataset. Our method demonstrates significant improvements over a strong baseline and achieves performance comparable to models trained with text.

View on arXiv
@article{zhang2025_2505.15333,
  title={ Leveraging Unit Language Guidance to Advance Speech Modeling in Textless Speech-to-Speech Translation },
  author={ Yuhao Zhang and Xiangnan Ma and Kaiqi Kou and Peizhuo Liu and Weiqiao Shan and Benyou Wang and Tong Xiao and Yuxin Huang and Zhengtao Yu and Jingbo Zhu },
  journal={arXiv preprint arXiv:2505.15333},
  year={ 2025 }
}
Comments on this paper

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.