Intelligible Lip-to-Speech Synthesis with Speech Units

31 May 2023

Papers citing "Intelligible Lip-to-Speech Synthesis with Speech Units"

23 / 23 papers shown

Title
LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models Danilo de Oliveira Julius Richter Tal Peer Timo Germann DiffM 19 0 0 16 May 2025
AlignDiT: Multimodal Aligned Diffusion Transformer for Synchronized Speech Generation J. Choi Ji-Hoon Kim Kim Sung-Bin Tae-Hyun Oh Joon Son Chung DiffM 49 0 0 29 Apr 2025
Supervising 3D Talking Head Avatars with Analysis-by-Audio-Synthesis Radek Daněček Carolin Schmitt Senya Polikovsky Michael J. Black 36 0 0 18 Apr 2025
VoiceCraft-Dub: Automated Video Dubbing with Neural Codec Language Models Kim Sung-Bin Jeongsoo Choi Puyuan Peng Joon Son Chung Tae-Hyun Oh David Harwath VGen 47 1 0 03 Apr 2025
From Faces to Voices: Learning Hierarchical Representations for High-quality Video-to-Speech Ji-Hoon Kim Jeongsoo Choi Jaehun Kim Chaeyoung Jung Joon Son Chung CVBM 53 1 0 21 Mar 2025
DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility Yifan Liu Yu Fang Zhouhan Lin 42 0 0 07 Mar 2025
NaturalL2S: End-to-End High-quality Multispeaker Lip-to-Speech Synthesis with Differential Digital Signal Processing Yifan Liang Fangkun Liu Andong Li Xiaodong Li C. Zheng 49 1 0 17 Feb 2025
Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction Zhaoxi Mu Xinyu Yang 40 5 0 19 Apr 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 27 5 0 18 Jan 2024
AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation J. Choi Se Jin Park Minsu Kim Y. Ro 33 12 0 05 Dec 2023
Multi-resolution HuBERT: Multi-resolution Speech Self-Supervised Learning with Masked Unit Prediction Jiatong Shi Hirofumi Inaguma Xutai Ma Ilia Kulikov Anna Y. Sun 48 24 0 04 Oct 2023
Visual Speech Recognition for Languages with Limited Labeled Data using Automatic Labels from Whisper Jeong Hun Yeo Minsu Kim Shinji Watanabe Y. Ro VLM 31 12 0 15 Sep 2023
Towards Practical and Efficient Image-to-Speech Captioning with Vision-Language Pre-training and Multi-modal Tokens Minsu Kim J. Choi Soumi Maiti Jeong Hun Yeo Shinji Watanabe Y. Ro VLM 26 6 0 15 Sep 2023
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge Minsu Kim Jeong Hun Yeo J. Choi Y. Ro 34 16 0 18 Aug 2023
DiffV2S: Diffusion-based Video-to-Speech Synthesis with Vision-guided Speaker Embedding J. Choi Joanna Hong Y. Ro DiffM 29 19 0 15 Aug 2023
Many-to-Many Spoken Language Translation via Unified Speech and Text Representation Learning with Unit-to-Unit Translation Minsu Kim J. Choi Dahun Kim Y. Ro 40 12 0 03 Aug 2023
RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations Neha Sahipjohn Neil Shah Vishal Tambrahalli Vineet Gandhi 19 2 0 03 Jul 2023
LipVoicer: Generating Speech from Silent Videos Guided by Lip Reading Yochai Yemini Aviv Shamsian Lior Bracha Sharon Gannot Ethan Fetaya DiffM 27 11 0 05 Jun 2023
End-to-end Audio-visual Speech Recognition with Conformers Pingchuan Ma Stavros Petridis M. Pantic 84 225 0 12 Feb 2021
Generative Spoken Language Modeling from Raw Audio Kushal Lakhotia Evgeny Kharitonov Wei-Ning Hsu Yossi Adi Adam Polyak ... Tu Nguyen Jade Copet Alexei Baevski A. Mohamed Emmanuel Dupoux AuLLM 191 337 0 01 Feb 2021
Lipreading using Temporal Convolutional Networks Brais Martínez Pingchuan Ma Stavros Petridis M. Pantic 168 239 0 23 Jan 2020
VoxCeleb2: Deep Speaker Recognition Joon Son Chung Arsha Nagrani Andrew Zisserman 251 2,233 0 14 Jun 2018
Lip Reading Sentences in the Wild Joon Son Chung A. Senior Oriol Vinyals Andrew Zisserman 167 784 0 16 Nov 2016