Text-to-Audio Generation Synchronized with Videos

8 March 2024

Shentong Mo

Jing Shi

Papers citing "Text-to-Audio Generation Synchronized with Videos"

32 / 32 papers shown

Title
Hearing from Silence: Reasoning Audio Descriptions from Silent Videos via Vision-Language Model Yong Ren Chenxing Li Le Xu Hao Gu Duzhen Zhang Yujie Chen Manjie Xu Ruibo Fu Shan Yang Dong Yu LRM 61 0 0 19 May 2025
DiffGAP: A Lightweight Diffusion Module in Contrastive Space for Bridging Cross-Model Gap Shentong Mo Zehua Chen Fan Bao Jun-Jie Zhu DiffM 97 1 0 15 Mar 2025
MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis Ho Kei Cheng Masato Ishii Akio Hayakawa Takashi Shibuya Alex Schwing Yuki Mitsufuji VGen 268 17 0 19 Dec 2024
STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment Yong Ren Chenxing Li Manjie Xu Wei Liang Yu Gu Rilin Chen Dong Yu VGen DiffM 83 9 0 13 Sep 2024
Video-to-Audio Generation with Hidden Alignment Manjie Xu Chenxing Li Yong Ren Rilin Chen Yu Gu Yu Gu Dong Yu Dong Yu DiffM VGen 81 12 0 10 Jul 2024
Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling Shentong Mo Pedro Morgado 64 14 0 02 Dec 2023
Weakly-Supervised Audio-Visual Segmentation Shentong Mo Bhiksha Raj VOS 77 13 0 25 Nov 2023
A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition Shentong Mo Pedro Morgado 67 22 0 30 May 2023
Adding Conditional Control to Text-to-Image Diffusion Models Lvmin Zhang Anyi Rao Maneesh Agrawala AI4CE 180 4,168 1 10 Feb 2023
MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation Ludan Ruan Yi Ma Huan Yang Huiguo He Bei Liu Jianlong Fu Nicholas Jing Yuan Qin Jin B. Guo DiffM VGen 109 189 0 19 Dec 2022
Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation Yusong Wu Kai Chen Tianyu Zhang Yuchen Hui Marianna Nezhurina Taylor Berg-Kirkpatrick Shlomo Dubnov CLIP 129 537 0 12 Nov 2022
Imagen Video: High Definition Video Generation with Diffusion Models Jonathan Ho William Chan Chitwan Saharia Jay Whang Ruiqi Gao ... Diederik P. Kingma Ben Poole Mohammad Norouzi David J. Fleet Tim Salimans VGen 165 1,541 0 05 Oct 2022
AudioGen: Textually Guided Audio Generation Felix Kreuk Gabriel Synnaeve Adam Polyak Uriel Singer Alexandre Défossez Jade Copet Devi Parikh Yaniv Taigman Yossi Adi DiffM 99 309 0 30 Sep 2022
A Closer Look at Weakly-Supervised Audio-Visual Source Localization Shentong Mo Pedro Morgado 137 66 0 30 Aug 2022
Diffsound: Discrete Diffusion Model for Text-to-sound Generation Dongchao Yang Jianwei Yu Helin Wang Wen Wang Chao Weng Yuexian Zou Dong Yu DiffM 90 305 0 20 Jul 2022
Taming Visually Guided Sound Generation Vladimir E. Iashin Esa Rahtu VLM 97 128 0 17 Oct 2021
Robust Audio-Visual Instance Discrimination Pedro Morgado Ishan Misra Nuno Vasconcelos SSL 87 110 0 29 Mar 2021
Score-Based Generative Modeling through Stochastic Differential Equations Yang Song Jascha Narain Sohl-Dickstein Diederik P. Kingma Abhishek Kumar Stefano Ermon Ben Poole DiffM SyDa 353 6,566 0 26 Nov 2020
Learning Representations from Audio-Visual Spatial Alignment Pedro Morgado Yi Li Nuno Vasconcelos SSL 77 123 0 03 Nov 2020
DiffWave: A Versatile Diffusion Model for Audio Synthesis Zhifeng Kong Ming-Yu Liu Jiaji Huang Kexin Zhao Bryan Catanzaro DiffM BDL 158 1,468 0 21 Sep 2020
Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing Yapeng Tian Dingzeyu Li Chenliang Xu 108 184 0 21 Jul 2020
VGGSound: A Large-scale Audio-Visual Dataset Honglie Chen Weidi Xie Andrea Vedaldi Andrew Zisserman 89 583 0 29 Apr 2020
Audio-Visual Instance Discrimination with Cross-Modal Agreement Pedro Morgado Nuno Vasconcelos Ishan Misra SSL 82 276 0 27 Apr 2020
Music Gesture for Visual Sound Separation Chuang Gan Deng Huang Hang Zhao J. Tenenbaum Antonio Torralba 95 205 0 20 Apr 2020
The Sound of Motions Hang Zhao Chuang Gan Wei-Chiu Ma Antonio Torralba 83 254 0 11 Apr 2019
Dual-modality seq2seq network for audio-visual event localization Yan-Bo Lin Yu-Jhe Li Y. Wang 64 128 0 20 Feb 2019
Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization Bruno Korbar Du Tran Lorenzo Torresani 99 476 0 30 Jun 2018
The Sound of Pixels Hang Zhao Chuang Gan Andrew Rouditchenko Carl Vondrick Josh H. McDermott Antonio Torralba VLM 102 536 0 09 Apr 2018
Learning to Localize Sound Source in Visual Scenes Arda Senocak Tae-Hyun Oh Junsik Kim Ming-Hsuan Yang In So Kweon SSL 66 345 0 10 Mar 2018
Look, Listen and Learn Relja Arandjelović Andrew Zisserman SSL 125 906 0 23 May 2017
CNN Architectures for Large-Scale Audio Classification Shawn Hershey Sourish Chaudhuri D. Ellis J. Gemmeke A. Jansen ... Rif A. Saurous Bryan Seybold M. Slaney Ron J. Weiss K. Wilson 128 2,507 0 29 Sep 2016
Efficient Estimation of Word Representations in Vector Space Tomas Mikolov Kai Chen G. Corrado J. Dean 3DV 684 31,544 0 16 Jan 2013