Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

5 January 2022

Papers citing "Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction"

50 / 207 papers shown

Title
DCIM-AVSR : Efficient Audio-Visual Speech Recognition via Dual Conformer Interaction Module Xinyu Wang Qian Wang Haolin Huang Yu Fang Mengjie Xu Qian Wang 31 0 0 31 Aug 2024
Masked Image Modeling: A Survey Vlad Hondru Florinel-Alin Croitoru Shervin Minaee Radu Tudor Ionescu N. Sebe 72 6 0 13 Aug 2024
Enhancing Dialogue Speech Recognition with Robust Contextual Awareness via Noise Representation Learning Wonjun Lee San Kim Gary Geunbae Lee 49 0 0 12 Aug 2024
Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing Jiarui Xie Mutahar Safdar Lequn Chen Seung Ki Moon Y. Zhao 45 1 0 09 Aug 2024
SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data Yichen Lu Álvaro Huertas-García Xuankai Chang Hengwei Bian Soumi Maiti Shinji Watanabe 46 2 0 01 Aug 2024
RAVSS: Robust Audio-Visual Speech Separation in Multi-Speaker Scenarios with Missing Visual Cues Tianrui Pan Jie Liu Bohan Wang Jie Tang Gangshan Wu 40 2 0 27 Jul 2024
dMel: Speech Tokenization made Simple Richard He Bai Tatiana Likhomanenko Ruixiang Zhang Zijin Gu Zakaria Aldeneh Navdeep Jaitly 40 4 0 22 Jul 2024
MuTT: A Multimodal Trajectory Transformer for Robot Skills Claudius Kienle Benjamin Alt Onur Celik P. Becker Darko Katic Rainer Jäkel Gerhard Neumann 38 2 0 22 Jul 2024
Tailored Design of Audio-Visual Speech Recognition Models using Branchformers David Gimeno-Gómez Carlos David Martínez Hinarejos 96 2 0 09 Jul 2024
ADAPT: Multimodal Learning for Detecting Physiological Changes under Missing Modalities Julie Mordacq Léo Milecki Maria Vakalopoulou Steve Oudot Vicky Kalogeiton OffRL MedIm 37 3 0 04 Jul 2024
Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition Sungnyun Kim Kangwook Jang Sangmin Bae Hoirin Kim Se-Young Yun 50 3 0 04 Jul 2024
MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization Adriana Fernandez-Lopez Honglie Chen Pingchuan Ma Lu Yin Q. Xiao Stavros Petridis Shiwei Liu Maja Pantic 46 2 0 25 Jun 2024
SyncVSR: Data-Efficient Visual Speech Recognition with End-to-End Crossmodal Audio Token Synchronization Young Jin Ahn Jungwoo Park Sangha Park Jonghyun Choi Kee-Eung Kim 34 7 0 18 Jun 2024
FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching Chaeyoung Jung Suyeon Lee Ji-Hoon Kim Joon Son Chung DiffM 47 4 0 13 Jun 2024
DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing Neha Sahipjohn Ashishkumar Gudmalwar Nirmesh Shah Pankaj Wasnik R. Shah 43 5 0 13 Jun 2024
CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction Xueyuan Chen Dongchao Yang Dingdong Wang Xixin Wu Zhiyong Wu Helen Meng 48 1 0 12 Jun 2024
Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation Se Jin Park Chae Won Kim Hyeongseop Rha Minsu Kim Joanna Hong Jeong Hun Yeo Yong Man Ro CVBM AuLLM 48 6 0 12 Jun 2024
Zero-Shot Fake Video Detection by Audio-Visual Consistency Xiaolou Li Zehua Liu Chen Chen Lantian Li Li Guo D. Wang 63 4 0 12 Jun 2024
MaLa-ASR: Multimedia-Assisted LLM-Based ASR Guanrou Yang Ziyang Ma Fan Yu Zhifu Gao Shiliang Zhang Xie Chen AuLLM 41 2 0 09 Jun 2024
LipGER: Visually-Conditioned Generative Error Correction for Robust Automatic Speech Recognition Sreyan Ghosh Sonal Kumar Ashish Seth Purva Chiniya Utkarsh Tyagi R. Duraiswami Dinesh Manocha 46 0 0 06 Jun 2024
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling Zeyue Tian Zhaoyang Liu Ruibin Yuan Jiahao Pan Xiaoqiang Huang Xu Tan Xu Tan Qifeng Chen Y. Guo VGen 104 16 0 06 Jun 2024
Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approach Mahsa Kadkhodaei Elyaderani Shahram Shirani 31 0 0 02 Jun 2024
ViSpeR: Multilingual Audio-Visual Speech Recognition Sanath Narayan Y. A. D. Djilali Ankit Singh Eustache Le Bihan Hakim Hacid VLM 33 0 0 27 May 2024
OpFlowTalker: Realistic and Natural Talking Face Generation via Optical Flow Guidance Shuheng Ge Haoyu Xing Li Zhang Xiangqian Wu 39 0 0 23 May 2024
Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation Dogucan Yaman Fevziye Irem Eyiokur Leonard Barmann Seymanur Akti H. K. Ekenel Alexander H. Waibel EGVM 33 9 0 07 May 2024
Unmasking Illusions: Understanding Human Perception of Audiovisual Deepfakes Ammarah Hashmi Sahibzada Adil Shahzad Chia-Wen Lin Yu Tsao Hsin-Min Wang 46 3 0 07 May 2024
Audio-Visual Target Speaker Extraction with Reverse Selective Auditory Attention Ruijie Tao Xinyuan Qian Yidi Jiang Junjie Li Jiadong Wang Haizhou Li 34 1 0 29 Apr 2024
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition Peihao Xiang Chaohao Lin Kaida Wu Ou Bai 34 3 0 28 Apr 2024
A review of deep learning-based information fusion techniques for multimodal medical image classification Yi-Hsuan Li Mostafa EL HABIB DAHO Pierre-Henri Conze Rachid Zeghlache Hugo Le Boité R. Tadayoni B. Cochener M. Lamard G. Quellec 38 31 0 23 Apr 2024
Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction Zhaoxi Mu Xinyu Yang 40 5 0 19 Apr 2024
SoundingActions: Learning How Actions Sound from Narrated Egocentric Videos Changan Chen Kumar Ashutosh Rohit Girdhar David Harwath Kristen Grauman EgoV SSL 28 6 0 08 Apr 2024
BRAVEn: Improving Self-Supervised Pre-training for Visual and Auditory Speech Recognition A. Haliassos Andreas Zinonos Rodrigo Mira Stavros Petridis Maja Pantic VLM SSL AI4TS 47 12 0 02 Apr 2024
Siamese Vision Transformers are Scalable Audio-visual Learners Yan-Bo Lin Gedas Bertasius 37 5 0 28 Mar 2024
Target Speech Extraction with Pre-trained AV-HuBERT and Mask-And-Recover Strategy Wenxuan Wu Xueyuan Chen Xixin Wu Haizhou Li Helen M. Meng 34 1 0 24 Mar 2024
XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception HyoJung Han Mohamed Anwar J. Pino Wei-Ning Hsu Marine Carpuat Bowen Shi Changhan Wang VLM 37 9 0 21 Mar 2024
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer Maxime Burchi Krishna C. Puvvada Jagadeesh Balam Boris Ginsburg Radu Timofte 44 8 0 14 Mar 2024
A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition Yusheng Dai Hang Chen Jun Du Ruoyu Wang Shihao Chen Jie Ma Haotian Wang Chin-Hui Lee 45 4 0 07 Mar 2024
FaceChain-ImagineID: Freely Crafting High-Fidelity Diverse Talking Faces from Disentangled Audio Chao Xu Yang Liu Jiazheng Xing Weida Wang Mingze Sun ... Tianxin Huang Siyuan Li Zhi-Qi Cheng Ying Tai Baigui Sun CVBM 54 11 0 04 Mar 2024
Towards Accurate Lip-to-Speech Synthesis in-the-Wild Sindhu B. Hegde Rudrabha Mukhopadhyay C. V. Jawahar Vinay P. Namboodiri 27 4 0 02 Mar 2024
Where Visual Speech Meets Language: VSP-LLM Framework for Efficient and Context-Aware Visual Speech Processing Jeong Hun Yeo Seunghee Han Minsu Kim Y. Ro 56 32 0 23 Feb 2024
AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies José-M. Acosta-Triana David Gimeno-Gómez Carlos David Martínez Hinarejos VLM VGen 47 2 0 20 Feb 2024
Comparison of Conventional Hybrid and CTC/Attention Decoders for Continuous Visual Speech Recognition David Gimeno-Gómez Carlos David Martínez Hinarejos 32 1 0 20 Feb 2024
Integrating Self-supervised Speech Model with Pseudo Word-level Targets from Visually-grounded Speech Model Hung-Chieh Fang Nai-Xuan Ye Yi-Jen Shih Puyuan Peng Hsuan-Fu Wang Layne Berry Hung-yi Lee David Harwath VLM 45 1 0 08 Feb 2024
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang 36 19 0 08 Feb 2024
Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction Xueyuan Chen Yuejiao Wang Xixin Wu Disong Wang Zhiyong Wu Xunying Liu Helen M. Meng 42 6 0 31 Jan 2024
Dynamic Transformer Architecture for Continual Learning of Multimodal Tasks Yuliang Cai Mohammad Rostami 33 4 0 27 Jan 2024
Efficient Training for Multilingual Visual Speech Recognition: Pre-training with Discretized Visual Speech Representation Minsu Kim Jeong Hun Yeo Se Jin Park J. Choi Y. Ro 27 5 0 18 Jan 2024
SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition Hao Wang Shuhei Kurita Shuichiro Shimizu Daisuke Kawahara 13 3 0 18 Jan 2024
HiCMAE: Hierarchical Contrastive Masked Autoencoder for Self-Supervised Audio-Visual Emotion Recognition Guoying Zhao Zheng Lian Bin Liu Jianhua Tao 53 29 0 11 Jan 2024
Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation Qiu-shi Zhu Jie Zhang Yu Gu Yuli Hu Lirong Dai SSL 43 11 0 07 Jan 2024