End-to-end Audio-visual Speech Recognition with Conformers

12 February 2021

Papers citing "End-to-end Audio-visual Speech Recognition with Conformers"

42 / 42 papers shown

Title
LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models Danilo de Oliveira Julius Richter Tal Peer Timo Germann DiffM 17 0 0 16 May 2025
SwinLip: An Efficient Visual Speech Encoder for Lip Reading Using Swin Transformer Young-Hu Park R.-H. Park Hyung-Min Park 49 0 0 07 May 2025
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Andrew Rouditchenko Saurabhchand Bhati Samuel Thomas Hilde Kuehne Rogerio Feris 113 1 0 03 Feb 2025
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim Sungwoo Cho Sangmin Bae Kangwook Jang Se-Young Yun SSL 68 1 0 23 Jan 2025
Uncovering the Visual Contribution in Audio-Visual Speech Recognition Zhaofeng Lin Naomi Harte 78 1 0 20 Jan 2025
Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language Jeong Hun Yeo Chae Won Kim Hyunjun Kim Hyeongseop Rha Seunghee Han Wen-Huang Cheng Y. Ro 52 3 0 03 Jan 2025
EmoDubber: Towards High Quality and Emotion Controllable Movie Dubbing Gaoxiang Cong Jiadong Pan Liang-Sheng Li Yuankai Qi Yuxin Peng A. Hengel Jian Yang Qingming Huang 92 6 0 12 Dec 2024
Large Language Models are Strong Audio-Visual Speech Recognition Learners Umberto Cappellazzo Minsu Kim Honglie Chen Pingchuan Ma Stavros Petridis Daniele Falavigna Alessio Brutti Maja Pantic 36 9 0 18 Sep 2024
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy? Yiwen Guan V. Trinh Vivek Voleti Jacob Whitehill 34 1 0 13 Sep 2024
Tailored Design of Audio-Visual Speech Recognition Models using Branchformers David Gimeno-Gómez Carlos David Martínez Hinarejos 88 2 0 09 Jul 2024
MaLa-ASR: Multimedia-Assisted LLM-Based ASR Guanrou Yang Ziyang Ma Fan Yu Zhifu Gao Shiliang Zhang Xie Chen AuLLM 36 2 0 09 Jun 2024
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models David Kurzendörfer Otniel-Bogdan Mercea A. Sophia Koepke Zeynep Akata VLM CLIP 31 2 0 09 Apr 2024
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer Maxime Burchi Krishna C. Puvvada Jagadeesh Balam Boris Ginsburg Radu Timofte 42 8 0 14 Mar 2024
Do VSR Models Generalize Beyond LRS3? Y. A. D. Djilali Sanath Narayan Eustache Le Bihan Haithem Boussaid Ebtesam Almazrouei Merouane Debbah 30 4 0 23 Nov 2023
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge Minsu Kim Jeong Hun Yeo J. Choi Y. Ro 34 16 0 18 Aug 2023
Audio-visual video-to-speech synthesis with synthesized input audio Triantafyllos Kefalas Yannis Panagakis M. Pantic VGen DiffM 30 1 0 31 Jul 2023
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Cheng Chen Chengwei Qin Qiu-shi Zhu E. Chng 29 5 0 18 Jun 2023
Multi-Temporal Lip-Audio Memory for Visual Speech Recognition Jeong Hun Yeo Minsu Kim Y. Ro 27 11 0 08 May 2023
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision Xubo Liu Egor Lakomkin Konstantinos Vougioukas Pingchuan Ma Honglie Chen ... Niko Moritz J. Kolár Stavros Petridis M. Pantic Christian Fuegen 44 19 0 30 Mar 2023
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert Jiadong Wang Xinyuan Qian Malu Zhang R. Tan Haizhou Li EGVM 22 93 0 29 Mar 2023
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR Paul Hongsuck Seo Arsha Nagrani Cordelia Schmid 27 15 0 29 Mar 2023
The DKU Post-Challenge Audio-Visual Wake Word Spotting System for the 2021 MISP Challenge: Deep Analysis Haoxu Wang Ming Cheng Qiang Fu Ming Li 31 8 0 04 Mar 2023
A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset J. Peymanfard Samin Heydarian Ali Lashini Hossein Zeinali Mohammad Reza Mohammadi N. Mozayani 21 10 0 21 Jan 2023
OLKAVS: An Open Large-Scale Korean Audio-Visual Speech Dataset J. Park Jung-Wook Hwang Kwanghee Choi Seung-Hyun Lee Jun-Hwan Ahn R.-H. Park Hyung-Min Park 21 3 0 16 Jan 2023
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement Wei-Ning Hsu Tal Remez Bowen Shi Jacob Donley Yossi Adi DiffM 27 11 0 21 Dec 2022
Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning Chen Chen Yuchen Hu Qiang Zhang Heqing Zou Beier Zhu E. Chng 25 26 0 10 Dec 2022
AVATAR submission to the Ego4D AV Transcription Challenge Paul Hongsuck Seo Arsha Nagrani Cordelia Schmid 14 0 0 18 Nov 2022
Streaming Audio-Visual Speech Recognition with Alignment Regularization Pingchuan Ma Niko Moritz Stavros Petridis Christian Fuegen M. Pantic 37 2 0 03 Nov 2022
Relaxed Attention for Transformer Models Timo Lohrenz Björn Möller Zhengyang Li Tim Fingscheidt KELM 29 11 0 20 Sep 2022
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality Wei-Ning Hsu Bowen Shi SSL VLM 19 41 0 14 Jul 2022
Visual Context-driven Audio Feature Enhancement for Robust End-to-End Audio-Visual Speech Recognition Joanna Hong Minsu Kim Daehun Yoo Y. Ro 26 20 0 13 Jul 2022
Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car Commands Wenliang Dai Samuel Cahyawijaya Tiezheng Yu Elham J. Barezi Pascale Fung 16 1 0 06 Jul 2022
Is Lip Region-of-Interest Sufficient for Lipreading? Jing-Xuan Zhang Genshun Wan Jia-Yu Pan 24 6 0 28 May 2022
Improving Multimodal Speech Recognition by Data Augmentation and Speech Representations Dan Oneaţă H. Cucu 19 19 0 27 Apr 2022
VocaLiST: An Audio-Visual Synchronisation Model for Lips and Voices V. S. Kadandale Juan F. Montesinos G. Haro 21 23 0 05 Apr 2022
Learning Contextually Fused Audio-visual Representations for Audio-visual Speech Recognition Zitian Zhang Jie M. Zhang Jian-Shu Zhang Ming Wu Xin Fang Lirong Dai SSL 35 10 0 15 Feb 2022
Transformer-Based Video Front-Ends for Audio-Visual Speech Recognition for Single and Multi-Person Video Dmitriy Serdyuk Otavio Braga Olivier Siohan ViT 89 40 0 25 Jan 2022
CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition Wenliang Dai Samuel Cahyawijaya Tiezheng Yu Elham J. Barezi Peng-Tao Xu ... Genta Indra Winata Qifeng Chen Xiaojuan Ma Bertram E. Shi Pascale Fung 41 11 0 11 Jan 2022
Robust Self-Supervised Audio-Visual Speech Recognition Bowen Shi Wei-Ning Hsu Abdel-rahman Mohamed 31 90 0 05 Jan 2022
LiRA: Learning Visual Speech Representations from Audio through Self-supervision Pingchuan Ma Rodrigo Mira Stavros Petridis Björn W. Schuller M. Pantic SSL 21 53 0 16 Jun 2021
Lipreading using Temporal Convolutional Networks Brais Martínez Pingchuan Ma Stavros Petridis M. Pantic 168 239 0 23 Jan 2020
Lip Reading Sentences in the Wild Joon Son Chung A. Senior Oriol Vinyals Andrew Zisserman 162 784 0 16 Nov 2016