Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

5 January 2022

Papers citing "Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction"

50 / 61 papers shown

Title
LipDiffuser: Lip-to-Speech Generation with Conditional Diffusion Models Danilo de Oliveira Julius Richter Tal Peer Timo Germann DiffM 19 0 0 16 May 2025
CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization Detao Bai Zhiheng Ma Xihan Wei Liefeng Bo 129 0 0 06 May 2025
mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Andrew Rouditchenko Saurabhchand Bhati Samuel Thomas Hilde Kuehne Rogerio Feris 116 1 0 03 Feb 2025
Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation Sungnyun Kim Sungwoo Cho Sangmin Bae Kangwook Jang Se-Young Yun SSL 68 1 0 23 Jan 2025
Personalized Lip Reading: Adapting to Your Unique Lip Movements with Vision and Language Jeong Hun Yeo Chae Won Kim Hyunjun Kim Hyeongseop Rha Seunghee Han Wen-Huang Cheng Y. Ro 59 3 0 03 Jan 2025
DiFiC: Your Diffusion Model Holds the Secret to Fine-Grained Clustering Ruohong Yang Peng Hu Xi Peng Xiting Liu Yunfan Li 39 0 0 25 Dec 2024
DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search Lei Yang Shaoyang Xu Deyi Xiong 39 0 0 25 Dec 2024
Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning Dragos-Alexandru Boldisor Stefan Smeu Dan Oneaţă Elisabeta Oneata 103 1 0 29 Nov 2024
The Sound of Water: Inferring Physical Properties from Pouring Liquids Piyush Bagad Makarand Tapaswi Cees G. M. Snoek Andrew Zisserman 45 0 0 18 Nov 2024
DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization C. Koutlis Symeon Papadopoulos 58 2 0 15 Nov 2024
Diffusion-based Unsupervised Audio-visual Speech Enhancement Jean-Eudes Ayilo Mostafa Sadeghi Romain Serizel Xavier Alameda-Pineda DiffM 20 0 0 04 Oct 2024
Measuring Sound Symbolism in Audio-visual Models Wei-Cheng Tseng Yi-Jen Shih David Harwath Raymond Mooney 37 0 0 18 Sep 2024
Large Language Models are Strong Audio-Visual Speech Recognition Learners Umberto Cappellazzo Minsu Kim Honglie Chen Pingchuan Ma Stavros Petridis Daniele Falavigna Alessio Brutti Maja Pantic 36 9 0 18 Sep 2024
Towards Global Localization using Multi-Modal Object-Instance Re-Identification Aneesh Chavan Vaibhav Agrawal Vineeth Bhat Sarthak Chittawar Siddharth Srivastava Chetan Arora K. M. Krishna 95 0 0 18 Sep 2024
Multi-modal Speech Transformer Decoders: When Do Multiple Modalities Improve Accuracy? Yiwen Guan V. Trinh Vivek Voleti Jacob Whitehill 42 1 0 13 Sep 2024
Masked Image Modeling: A Survey Vlad Hondru Florinel-Alin Croitoru Shervin Minaee Radu Tudor Ionescu N. Sebe 72 6 0 13 Aug 2024
Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing Jiarui Xie Mutahar Safdar Lequn Chen Seung Ki Moon Y. Zhao 43 1 0 09 Aug 2024
Tailored Design of Audio-Visual Speech Recognition Models using Branchformers David Gimeno-Gómez Carlos David Martínez Hinarejos 94 2 0 09 Jul 2024
FlowAVSE: Efficient Audio-Visual Speech Enhancement with Conditional Flow Matching Chaeyoung Jung Suyeon Lee Ji-Hoon Kim Joon Son Chung DiffM 47 4 0 13 Jun 2024
MaLa-ASR: Multimedia-Assisted LLM-Based ASR Guanrou Yang Ziyang Ma Fan Yu Zhifu Gao Shiliang Zhang Xie Chen AuLLM 41 2 0 09 Jun 2024
VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modeling Zeyue Tian Zhaoyang Liu Ruibin Yuan Jiahao Pan Xiaoqiang Huang Xu Tan Xu Tan Qifeng Chen Y. Guo VGen 102 16 0 06 Jun 2024
Unmasking Illusions: Understanding Human Perception of Audiovisual Deepfakes Ammarah Hashmi Sahibzada Adil Shahzad Chia-Wen Lin Yu Tsao Hsin-Min Wang 46 3 0 07 May 2024
MultiMAE-DER: Multimodal Masked Autoencoder for Dynamic Emotion Recognition Peihao Xiang Chaohao Lin Kaida Wu Ou Bai 34 3 0 28 Apr 2024
Siamese Vision Transformers are Scalable Audio-visual Learners Yan-Bo Lin Gedas Bertasius 37 5 0 28 Mar 2024
Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer Maxime Burchi Krishna C. Puvvada Jagadeesh Balam Boris Ginsburg Radu Timofte 44 8 0 14 Mar 2024
It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognition Chen Chen Ruizhe Li Yuchen Hu Sabato Marco Siniscalchi Pin-Yu Chen Ensiong Chng Chao-Han Huck Yang 33 19 0 08 Feb 2024
Neural Text to Articulate Talk: Deep Text to Audiovisual Speech Synthesis achieving both Auditory and Photo-realism Georgios Milis P. Filntisis A. Roussos Petros Maragos CVBM 34 2 0 11 Dec 2023
Do VSR Models Generalize Beyond LRS3? Y. A. D. Djilali Sanath Narayan Eustache Le Bihan Haithem Boussaid Ebtesam Almazrouei Merouane Debbah 35 4 0 23 Nov 2023
AVTENet: Audio-Visual Transformer-based Ensemble Network Exploiting Multiple Experts for Video Deepfake Detection Ammarah Hashmi Sahibzada Adil Shahzad Chia-Wen Lin Yu Tsao Hsin-Min Wang ViT 53 6 0 19 Oct 2023
Lip Reading for Low-resource Languages by Learning and Combining General Speech Knowledge and Language-specific Knowledge Minsu Kim Jeong Hun Yeo J. Choi Y. Ro 34 16 0 18 Aug 2023
Audio-visual video-to-speech synthesis with synthesized input audio Triantafyllos Kefalas Yannis Panagakis M. Pantic VGen DiffM 38 1 0 31 Jul 2023
Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognition Yuchen Hu Ruizhe Li Cheng Chen Chengwei Qin Qiu-shi Zhu E. Chng 31 5 0 18 Jun 2023
AV-TranSpeech: Audio-Visual Robust Speech-to-Speech Translation Rongjie Huang Huadai Liu Xize Cheng Yi Ren Lin Li ... Jinzheng He Lichao Zhang Jinglin Liu Xiaoyue Yin Zhou Zhao 72 8 0 24 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 48 115 0 18 May 2023
SynthVSR: Scaling Up Visual Speech Recognition With Synthetic Supervision Xubo Liu Egor Lakomkin Konstantinos Vougioukas Pingchuan Ma Honglie Chen ... Niko Moritz J. Kolár Stavros Petridis M. Pantic Christian Fuegen 52 19 0 30 Mar 2023
Seeing What You Said: Talking Face Generation Guided by a Lip Reading Expert Jiadong Wang Xinyuan Qian Malu Zhang R. Tan Haizhou Li EGVM 22 93 0 29 Mar 2023
Transformadores: Fundamentos teoricos y Aplicaciones J. D. L. Torre 78 0 0 18 Feb 2023
A Multi-Purpose Audio-Visual Corpus for Multi-Modal Persian Speech Recognition: the Arman-AV Dataset J. Peymanfard Samin Heydarian Ali Lashini Hossein Zeinali Mohammad Reza Mohammadi N. Mozayani 29 10 0 21 Jan 2023
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement Wei-Ning Hsu Tal Remez Bowen Shi Jacob Donley Yossi Adi DiffM 27 12 0 21 Dec 2022
Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning Chen Chen Yuchen Hu Qiang Zhang Heqing Zou Beier Zhu E. Chng 33 26 0 10 Dec 2022
Learning to Dub Movies via Hierarchical Prosody Models Gaoxiang Cong Liang Li Yuankai Qi Zhengjun Zha Qi Wu Wen-yu Wang Bin Jiang Ming Yang Qin Huang 75 25 0 08 Dec 2022
FakeOut: Leveraging Out-of-domain Self-supervision for Multi-modal Video Deepfake Detection Gil Knafo Ohad Fried 23 5 0 01 Dec 2022
Comparative layer-wise analysis of self-supervised speech models Ankita Pasad Bowen Shi Karen Livescu SSL 33 109 0 08 Nov 2022
Streaming Audio-Visual Speech Recognition with Alignment Regularization Pingchuan Ma Niko Moritz Stavros Petridis Christian Fuegen M. Pantic 37 2 0 03 Nov 2022
Clean Text and Full-Body Transformer: Microsoft's Submission to the WMT22 Shared Task on Sign Language Translation S. Dey Abhilash Pal Cyrine Chaabani Oscar Koller SLR 20 5 0 24 Oct 2022
TVLT: Textless Vision-Language Transformer Zineng Tang Jaemin Cho Yixin Nie Joey Tianyi Zhou VLM 51 28 0 28 Sep 2022
Relaxed Attention for Transformer Models Timo Lohrenz Björn Möller Zhengyang Li Tim Fingscheidt KELM 29 11 0 20 Sep 2022
Visual Speech-Aware Perceptual 3D Facial Expression Reconstruction from Videos P. Filntisis George Retsinas Foivos Paraperas-Papantoniou Athanasios Katsamanis A. Roussos Petros Maragos 3DH 18 27 0 22 Jul 2022
u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modality Wei-Ning Hsu Bowen Shi SSL VLM 27 41 0 14 Jul 2022
Multimodal Learning with Transformers: A Survey P. Xu Xiatian Zhu David A. Clifton ViT 72 527 0 13 Jun 2022