Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation

In this paper, we propose a method for annotating phonemic and prosodic labels on a given audio-transcript pair, aimed at constructing Japanese text-to-speech (TTS) datasets. Our approach involves fine-tuning a large-scale pre-trained automatic speech recognition (ASR) model, conditioned on ground truth transcripts, to simultaneously output phrase-level graphemes and annotation labels. To further correct errors in phonemic labeling, we employ a decoding strategy that utilizes dictionary prior knowledge. The objective evaluation results demonstrate that our proposed method outperforms previous approaches relying solely on text or audio. The subjective evaluation results indicate that the naturalness of speech synthesized by the TTS model, trained with labels annotated using our method, is comparable to that of a model trained with manual annotations.

@article{hu2025_2506.07646,
  title={ Transcript-Prompted Whisper with Dictionary-Enhanced Decoding for Japanese Speech Annotation },
  author={ Rui Hu and Xiaolong Lin and Jiawang Liu and Shixi Huang and Zhenpeng Zhan },
  journal={arXiv preprint arXiv:2506.07646},
  year={ 2025 }
}