v1v2 (latest)

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

19 December 2024

Papers citing "MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis"

50 / 72 papers shown

Title
Sonic4D: Spatial Audio Generation for Immersive 4D Scene Exploration Siyi Xie Hanxin Zhu Tianyu He X. Li Zhibo Chen VGen 28 0 0 18 Jun 2025
Average Calibration Losses for Reliable Uncertainty in Medical Image Segmentation Theodore Barfoot Luis C. Garcia-Peraza-Herrera Samet Akcay Ben Glocker Tom Vercauteren UQCV 154 0 0 04 Jun 2025
AudioGenie: A Training-Free Multi-Agent Framework for Diverse Multimodality-to-Multiaudio Generation Yan Rong Jinting Wang Shan Yang Guangzhi Lei Li Liu DiffM VGen 81 0 0 28 May 2025
Towards Video to Piano Music Generation with Chain-of-Perform Support Benchmarks Chang Liu Haomin Zhang Shiyu Xia Zihao Chen Chaofan Ding Xin Yue Huizhe Chen Xinhan Di 58 0 0 26 May 2025
SpecMaskFoley: Steering Pretrained Spectral Masked Generative Transformer Toward Synchronized Video-to-audio Synthesis via ControlNet Zhi-Wei Zhong Akira Takahashi Shuyang Cui Keisuke Toyama Shusuke Takahashi Yuki Mitsufuji VGen 74 0 0 22 May 2025
OmniAudio: Generating Spatial Audio from 360-Degree Video Huadai Liu Tianyi Luo Qikai Jiang Kaicheng Luo Peiwen Sun ... Xin Li Shiliang Zhang Zhijie Yan Zhou Zhao Wei Xue VGen 121 1 0 21 Apr 2025
OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking Zhongjian Wang Peng Zhang Jinwei Qi Guangyuan Wang Sheng Xu Chaonan Ji Sheng Xu Bang Zhang Liefeng Bo DiffM VGen 146 0 0 03 Apr 2025
Enhance Generation Quality of Flow Matching V2A Model via Multi-Step CoT-Like Guidance and Combined Preference Optimization Haomin Zhang Siyang Song Haoyu Wang Zihao Chen Xianglong Liu Chaofan Ding Xinhan Di 91 0 0 28 Mar 2025
DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos Yunming Liang Zihao Chen Chaofan Ding Xinhan Di DiffM VGen 117 0 0 28 Mar 2025
DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation Haomin Zhang Chang Liu Junjie Zheng Zihao Chen Chaofan Ding Xinhan Di DiffM VGen 180 0 0 28 Mar 2025
Long-Video Audio Synthesis with Multi-Agent Collaboration Yehang Zhang Xinli Xu Xiaojie Xu L. Liu Yuxiao Chen DiffM VGen 111 1 0 13 Mar 2025
The Curse of Conditions: Analyzing and Improving Optimal Transport for Conditional Flow-Based Generation Ho Kei Cheng Alexander Schwing OT 116 1 0 13 Mar 2025
AudioX: Diffusion Transformer for Anything-to-Audio Generation Zeyue Tian Yizhu Jin Zhaoyang Liu Ruibin Yuan Xu Tan Qifeng Chen Wei Xue Yu Guo 120 6 0 13 Mar 2025
KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation Yoonjin Chung Pilsun Eu Junwon Lee Keunwoo Choi Juhan Nam Ben Sangbae Chon EGVM 107 4 0 21 Feb 2025
Gotta Hear Them All: Sound Source Aware Vision to Audio Generation Wei Guo Heng Wang Jianbo Ma Weidong Cai DiffM 185 5 0 23 Nov 2024
Temporally Aligned Audio for Video with Autoregression Ilpo Viertola Vladimir E. Iashin Esa Rahtu VGen 85 13 0 20 Sep 2024
Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis Zhiqi Huang Dan Luo Jun Wang Huan Liao Zhiheng Li Zhiyong Wu VGen 93 4 0 13 Sep 2024
STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment Yong Ren Chenxing Li Manjie Xu Wei Liang Yu Gu Rilin Chen Dong Yu VGen DiffM 103 9 0 13 Sep 2024
Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis Qi Yang Binjie Mao Zili Wang Xing Nie Pengfei Gao Ying Guo Cheng Zhen Pengfei Yan Shiming Xiang VGen DiffM 86 5 0 10 Sep 2024
Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Sound Junwon Lee Jaekwon Im Dabin Kim Juhan Nam VGen 142 10 0 21 Aug 2024
Masked Generative Video-to-Audio Transformers with Enhanced Synchronicity Santiago Pascual Chunghsin Yeh Ioannis Tsiamas Joan Serrà DiffM VGen 95 16 0 15 Jul 2024
Read, Watch and Scream! Sound Generation from Text and Video Yujin Jeong Yunji Kim Sanghyuk Chun Jiyoung Lee VGen DiffM 91 15 0 08 Jul 2024
FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Sounds Yiming Zhang Yicheng Gu Yanhong Zeng Zhening Xing Yuancheng Wang Zhizheng Wu Kai Chen VGen 111 41 0 01 Jul 2024
Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matching Yongqi Wang Wenxiang Guo Rongjie Huang Jia-Bin Huang Zehan Wang Fuming You Ruiqi Li Zhou Zhao VGen DiffM 162 13 0 01 Jun 2024
A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation Gwanghyun Kim Alonso Martinez Yu-Chuan Su Brendan Jou José Lezama ... Lijun Yu Lu Jiang A. Jansen Jacob Walker Krishna Somandepalli 79 9 0 22 May 2024
Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization Navonil Majumder Chia-Yu Hung Deepanway Ghosal Wei-Ning Hsu Rada Mihalcea Soujanya Poria 151 61 0 15 Apr 2024
Text-to-Audio Generation Synchronized with Videos Shentong Mo Jing Shi Yapeng Tian DiffM VGen 95 18 0 08 Mar 2024
Audio-Synchronized Visual Animation Lin Zhang Shentong Mo Yijing Zhang Pedro Morgado DiffM 104 20 0 08 Mar 2024
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Patrick Esser Sumith Kulal A. Blattmann Rahim Entezari Jonas Muller ... Zion English Kyle Lacey Alex Goodwin Yannik Marek Robin Rombach DiffM 327 1,410 0 05 Mar 2024
Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners Yazhou Xing Yin-Yin He Zeyue Tian Xintao Wang Qifeng Chen 122 57 0 27 Feb 2024
Synchformer: Efficient Synchronization from Sparse Cues Vladimir E. Iashin Weidi Xie Esa Rahtu Andrew Zisserman 98 14 0 29 Jan 2024
T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis Yoonjin Chung Junwon Lee Juhan Nam 102 15 0 17 Jan 2024
Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation Jinlong Xue Yayue Deng Yingming Gao Ya Li DiffM 105 36 0 02 Jan 2024
Analyzing and Improving the Training Dynamics of Diffusion Models Tero Karras M. Aittala J. Lehtinen Janne Hellsten Timo Aila S. Laine 153 204 0 05 Dec 2023
CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generation Zineng Tang Ziyi Yang Mahmoud Khademi Yang Liu Chenguang Zhu Mohit Bansal LRM MLLM AuLLM 137 52 0 30 Nov 2023
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment Bin Zhu Bin Lin Munan Ning Yang Yan Jiaxi Cui ... Zongwei Li Wancai Zhang Zhifeng Li Wei Liu Liejie Yuan VLM MLLM 211 229 0 03 Oct 2023
FoleyGen: Visually-Guided Audio Generation Xinhao Mei Varun K. Nagaraja Gaël Le Lan Zhaoheng Ni Ernie Chang Yangyang Shi Vikas Chandra VGen 88 24 0 19 Sep 2023
Natural Language Supervision for General-Purpose Audio Representations Benjamin Elizalde Soham Deshmukh Huaming Wang AuLLM AI4TS 92 59 0 11 Sep 2023
CoNeTTE: An efficient Audio Captioning system leveraging multiple datasets with Task Embedding Etienne Labbé Thomas Pellegrini J. Pinquier 86 14 0 01 Sep 2023
V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models Heng Wang Jianbo Ma Santiago Pascual Richard Cartwright Weidong (Tom) Cai VGen 118 43 0 18 Aug 2023
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining Haohe Liu Yiitan Yuan Xubo Liu Xinhao Mei Qiuqiang Kong Qiao Tian Yuping Wang Wenwu Wang Yuxuan Wang Mark D. Plumbley DiffM 138 247 0 10 Aug 2023
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models Simian Luo Chuanhao Yan Chenxu Hu Hang Zhao DiffM 105 83 0 29 Jun 2023
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation Jia-Bin Huang Yi Ren Rongjie Huang Dongchao Yang Zhenhui Ye Chen Zhang Jinglin Liu Xiang Yin Zejun Ma Zhou Zhao DiffM 123 64 0 29 May 2023
Any-to-Any Generation via Composable Diffusion Zineng Tang Ziyi Yang Chenguang Zhu Michael Zeng Joey Tianyi Zhou VGen DiffM 127 191 0 19 May 2023
ImageBind: One Embedding Space To Bind Them All Rohit Girdhar Alaaeldin El-Nouby Zhuang Liu Mannat Singh Kalyan Vasudev Alwala Armand Joulin Ishan Misra VLM 219 945 0 09 May 2023
Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model Deepanway Ghosal Navonil Majumder Ambuj Mehrish Soujanya Poria 237 152 0 24 Apr 2023
Conditional Generation of Audio from Video via Foley Analogies Yuexi Du Ziyang Chen Justin Salamon Bryan C. Russell Andrew Owens VGen 75 40 0 17 Apr 2023
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research Xinhao Mei Chutong Meng Haohe Liu Qiuqiang Kong Tom Ko Chengqi Zhao Mark D. Plumbley Yuexian Zou Wenwu Wang 184 220 0 30 Mar 2023
Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos Kun Su Kaizhi Qian Eli Shlizerman Antonio Torralba Chuang Gan VGen AI4CE 87 20 0 29 Mar 2023
Improving and generalizing flow-based generative models with minibatch optimal transport Alexander Tong Kilian Fatras Nikolay Malkin G. Huguet Yanlei Zhang Jarrid Rector-Brooks Guy Wolf Yoshua Bengio OOD DiffM OT 148 308 0 01 Feb 2023