Self-Supervised Audio-Visual Soundscape Stylization

22 September 2024

Tingle Li

Renhao Wang

Po-Yao Huang

Andrew Owens

Gopala Anumanchipalli

DiffM

SSL

ArXiv (abs)PDF HTML

Papers citing "Self-Supervised Audio-Visual Soundscape Stylization"

50 / 76 papers shown

Title
Generating Visual Scenes from Touch Fengyu Yang Jiacheng Zhang Andrew Owens DiffM 74 27 0 26 Sep 2023
Self-Supervised Visual Acoustic Matching Arjun Somayazulu Changan Chen Kristen Grauman SSL 66 12 0 27 Jul 2023
Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models Simian Luo Chuanhao Yan Chenxu Hu Hang Zhao DiffM 82 83 0 29 Jun 2023
ImageBind: One Embedding Space To Bind Them All Rohit Girdhar Alaaeldin El-Nouby Zhuang Liu Mannat Singh Kalyan Vasudev Alwala Armand Joulin Ishan Misra VLM 158 940 0 09 May 2023
On Uni-Modal Feature Learning in Supervised Multi-Modal Learning Chenzhuang Du Jiaye Teng Tingle Li Yichen Liu Tianyuan Yuan Yue Wang Yang Yuan Hang Zhao 168 45 0 02 May 2023
Conditional Generation of Audio from Video via Foley Analogies Yuexi Du Ziyang Chen Justin Salamon Bryan C. Russell Andrew Owens VGen 65 40 0 17 Apr 2023
AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models Yuancheng Wang Zeqian Ju Xuejiao Tan Lei He Zhizheng Wu Jiang Bian Sheng Zhao DiffM 125 55 0 03 Apr 2023
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research Xinhao Mei Chutong Meng Haohe Liu Qiuqiang Kong Tom Ko Chengqi Zhao Mark D. Plumbley Yuexian Zou Wenwu Wang 124 218 0 30 Mar 2023
Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation Ziyang Chen Shengyi Qian Andrew Owens 76 13 0 20 Mar 2023
Epic-Sounds: A Large-scale Dataset of Actions That Sound Jaesung Huh Jacob Chalk Evangelos Kazakos Dima Damen Andrew Zisserman EgoV 71 43 0 01 Feb 2023
SingSong: Generating musical accompaniments from singing Chris Donahue Antoine Caillon Adam Roberts Ethan Manilow P. Esling ... Mauro Verzetti Ian Simon Olivier Pietquin Neil Zeghidour Jesse Engel 80 55 0 30 Jan 2023
Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models Rongjie Huang Jia-Bin Huang Dongchao Yang Yi Ren Luping Liu Mingze Li Zhenhui Ye Jinglin Liu Xiaoyue Yin Zhou Zhao DiffM 219 343 0 30 Jan 2023
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models Haohe Liu Zehua Chen Yiitan Yuan Xinhao Mei Xubo Liu Danilo Mandic Wenwu Wang Mark D. Plumbley DiffM 147 506 0 29 Jan 2023
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection Chao Feng Ziyang Chen Andrew Owens 65 77 0 04 Jan 2023
Robust Speech Recognition via Large-Scale Weak Supervision Alec Radford Jong Wook Kim Tao Xu Greg Brockman C. McLeavey Ilya Sutskever OffRL 203 3,732 0 06 Dec 2022
Touch and Go: Learning from Human-Collected Vision and Touch Fengyu Yang Chenyang Ma Jiacheng Zhang Jing Zhu Wenzhen Yuan Andrew Owens 64 60 0 22 Nov 2022
InstructPix2Pix: Learning to Follow Image Editing Instructions Tim Brooks Aleksander Holynski Alexei A. Efros DiffM 209 1,830 0 17 Nov 2022
I Hear Your True Colors: Image Guided Audio Generation Roy Sheffer Yossi Adi VLM 62 76 0 06 Nov 2022
AudioGen: Textually Guided Audio Generation Felix Kreuk Gabriel Synnaeve Adam Polyak Uriel Singer Alexandre Défossez Jade Copet Devi Parikh Yaniv Taigman Yossi Adi DiffM 104 309 0 30 Sep 2022
Classifier-Free Diffusion Guidance Jonathan Ho Tim Salimans FaML 196 3,963 0 26 Jul 2022
Diffsound: Discrete Diffusion Model for Text-to-sound Generation Dongchao Yang Jianwei Yu Helin Wang Wen Wang Chao Weng Yuexian Zou Dong Yu DiffM 90 305 0 20 Jul 2022
Style Transfer of Audio Effects with Differentiable Signal Processing C. Steinmetz Nicholas J. Bryan Joshua D. Reiss 50 44 0 18 Jul 2022
Masked Autoencoders that Listen Po-Yao (Bernie) Huang Hu Xu Juncheng Billy Li Alexei Baevski Michael Auli Wojciech Galuba Florian Metze Christoph Feichtenhofer 100 286 0 13 Jul 2022
Learning Visual Styles from Audio-Visual Associations Tingle Li Yichen Liu Andrew Owens Hang Zhao DiffM 61 22 0 10 May 2022
Visual Acoustic Matching Changan Chen Ruohan Gao P. Calamia Kristen Grauman 64 57 0 14 Feb 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 555 4,409 0 28 Jan 2022
High-Resolution Image Synthesis with Latent Diffusion Models Robin Rombach A. Blattmann Dominik Lorenz Patrick Esser Bjorn Ommer 3DV 493 15,734 0 20 Dec 2021
Audio-Visual Synchronisation in the wild Honglie Chen Weidi Xie Triantafyllos Afouras Arsha Nagrani Andrea Vedaldi Andrew Zisserman 113 39 0 08 Dec 2021
Sound-Guided Semantic Image Manipulation Seung Hyun Lee Wonseok Roh Wonmin Byeon Sang Ho Yoon Chanyoung Kim Jinkyu Kim Sangpil Kim DiffM 93 43 0 30 Nov 2021
The Cocktail Fork Problem: Three-Stem Audio Separation for Real-World Soundtracks Darius Petermann Gordon Wichern Zhong-Qiu Wang Jonathan Le Roux 44 38 0 19 Oct 2021
Taming Visually Guided Sound Generation Vladimir E. Iashin Esa Rahtu VLM 97 128 0 17 Oct 2021
Neural Dubber: Dubbing for Videos According to Scripts Chenxu Hu Qiao Tian Tingle Li Yuping Wang Yuxuan Wang Hang Zhao DiffM VGen 77 43 0 15 Oct 2021
Localizing Visual Sounds the Hard Way Honglie Chen Weidi Xie Triantafyllos Afouras Arsha Nagrani Andrea Vedaldi Andrew Zisserman ObjD 85 190 0 06 Apr 2021
Image2Reverb: Cross-Modal Reverb Impulse Response Synthesis Nikhil Singh Jeff Mentch Jerry Ng Matthew Beveridge Iddo Drori 55 47 0 26 Mar 2021
Paint by Word A. Andonian David Bau Audrey Cui YeonHwan Park Ali Jahanian Antonio Torralba A. Oliva DiffM 73 125 0 19 Mar 2021
Space-Time Crop & Attend: Improving Cross-modal Video Representation Learning Mandela Patrick Yuki M. Asano Bernie Huang Ishan Misra Florian Metze Joao Henriques Andrea Vedaldi AI4TS 88 35 0 18 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 972 29,810 0 26 Feb 2021
Taming Transformers for High-Resolution Image Synthesis Patrick Esser Robin Rombach Bjorn Ommer ViT 131 2,999 0 17 Dec 2020
CVC: Contrastive Learning for Non-parallel Voice Conversion Tingle Li Yichen Liu Chenxu Hu Hang Zhao DRL 72 13 0 02 Nov 2020
HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis Jungil Kong Jaehyeon Kim Jaekyoung Bae 179 1,947 0 12 Oct 2020
Denoising Diffusion Implicit Models Jiaming Song Chenlin Meng Stefano Ermon VLM DiffM 289 7,469 0 06 Oct 2020
FSD50K: An Open Dataset of Human-Labeled Sound Events Eduardo Fonseca Xavier Favory Jordi Pons F. Font Xavier Serra 100 466 0 01 Oct 2020
Foley Music: Learning to Generate Music from Videos Chuang Gan Deng Huang Peihao Chen J. Tenenbaum Antonio Torralba VGen 49 139 0 21 Jul 2020
Denoising Diffusion Probabilistic Models Jonathan Ho Ajay Jain Pieter Abbeel DiffM 715 18,310 0 19 Jun 2020
Telling Left from Right: Learning Spatial Correspondence of Sight and Sound Karren D. Yang Bryan C. Russell Justin Salamon SSL 85 76 0 11 Jun 2020
Atss-Net: Target Speaker Separation via Attention-based Neural Network Tingle Li Qingjian Lin Yuanyuan Bao Ming Li 31 38 0 19 May 2020
Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis Prajwal K R Rudrabha Mukhopadhyay Vinay P. Namboodiri C. V. Jawahar 65 113 0 17 May 2020
VGGSound: A Large-scale Audio-Visual Dataset Honglie Chen Weidi Xie Andrea Vedaldi Andrew Zisserman 89 583 0 29 Apr 2020
Audio-Visual Instance Discrimination with Cross-Modal Agreement Pedro Morgado Nuno Vasconcelos Ishan Misra SSL 82 276 0 27 Apr 2020
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition Qiuqiang Kong Yin Cao Turab Iqbal Yuxuan Wang Wenwu Wang Mark D. Plumbley VLM SSL 194 1,084 0 21 Dec 2019