Continual Speech Learning with Fused Speech Features

Rapid growth in speech data demands adaptive models, as traditional static methods fail to keep pace with dynamic and diverse speech information. We introduce continuous speech learning, a new set-up targeting at bridging the adaptation gap in current speech models. We use the encoder-decoder Whisper model to standardize speech tasks into a generative format. We integrate a learnable gated-fusion layer on the top of the encoder to dynamically select task-specific features for downstream tasks. Our approach improves accuracy significantly over traditional methods in six speech processing tasks, demonstrating gains in adapting to new speech tasks without full retraining.

@article{wang2025_2506.01496,
  title={ Continual Speech Learning with Fused Speech Features },
  author={ Guitao Wang and Jinming Zhao and Hao Yang and Guilin Qi and Tongtong Wu and Gholamreza Haffari },
  journal={arXiv preprint arXiv:2506.01496},
  year={ 2025 }
}