v1v2 (latest)

VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset

29 May 2023

ArXiv (abs)PDF HTML Github (277★)

Papers citing "VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset"

29 / 79 papers shown

Title
Tarsier: Recipes for Training and Evaluating Large Video Description Models Jiawei Wang Liping Yuan Yuchen Zhang 110 67 0 30 Jun 2024
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models Guangzhi Sun Wenyi Yu Changli Tang Xianzhao Chen Tian Tan Wei Li Lu Lu Zejun Ma Yuxuan Wang Chao Zhang 95 35 0 22 Jun 2024
Explore the Limits of Omni-modal Pretraining at Scale Yiyuan Zhang Handong Li Jing Liu Xiangyu Yue VLM LRM 87 1 0 13 Jun 2024
2DP-2MRC: 2-Dimensional Pointer-based Machine Reading Comprehension Method for Multimodal Moment Retrieval Jiajun He Tomoki Toda 104 1 0 10 Jun 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 139 45 0 26 May 2024
From Sora What We Can See: A Survey of Text-to-Video Generation Rui Sun Yumin Zhang Tejal Shah Jiahao Sun Shuoying Zhang Wenqi Li Haoran Duan Bo Wei R. Ranjan EGVM 131 20 0 17 May 2024
SonifyAR: Context-Aware Sound Generation in Augmented Reality Xia Su Jon E. Froehlich Eunyee Koh Chang Xiao 58 3 0 11 May 2024
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition Shijian Deng Erin E. Kosloski Siddhi Patel Zeke A. Barnett Yiyang Nan ... William T. Doan Matthew Wang Harsh Singh P. Rollins Yapeng Tian 66 5 0 22 Mar 2024
InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding Yi Wang Kunchang Li Xinhao Li Jiashuo Yu Yinan He ... Hongjie Zhang Yifei Huang Yu Qiao Yali Wang Limin Wang 88 79 0 22 Mar 2024
Sora as an AGI World Model? A Complete Survey on Text-to-Video Generation Joseph Cho Fachrina Dewi Puspitasari Sheng Zheng Jingyao Zheng Lik-Hang Lee Tae-Ho Kim Choong Seon Hong Chaoning Zhang EGVM VGen 104 43 0 08 Mar 2024
CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios Qilang Ye Zitong Yu Rui Shao Xinyu Xie Philip Torr Xiaochun Cao MLLM 112 30 0 07 Mar 2024
VideoPrism: A Foundational Visual Encoder for Video Understanding Long Zhao N. B. Gundavarapu Liangzhe Yuan Hao Zhou Shen Yan ... Huisheng Wang Hartwig Adam Mikhail Sirotenko Ting Liu Boqing Gong VGen 123 36 0 20 Feb 2024
Real-World Robot Applications of Foundation Models: A Review Kento Kawaharazuka T. Matsushima Andrew Gambardella Jiaxian Guo Chris Paxton Andy Zeng OffRL VLM LM&Ro 116 54 0 08 Feb 2024
CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion Shoubin Yu Jaehong Yoon Mohit Bansal 175 7 0 08 Feb 2024
Detours for Navigating Instructional Videos Kumar Ashutosh Zihui Xue Tushar Nagarajan Kristen Grauman 127 6 0 03 Jan 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Chenliang Xu Jiebo Luo Chenliang Xu VLM 222 100 0 29 Dec 2023
Data-Efficient Multimodal Fusion on a Single GPU Noël Vouitsis Zhaoyan Liu S. Gorti Valentin Villecroze Jesse C. Cresswell Guangwei Yu Gabriel Loaiza-Ganem M. Volkovs 127 3 0 15 Dec 2023
A Video is Worth 10,000 Words: Training and Benchmarking with Diverse Captions for Better Long Video Retrieval M. Gwilliam Michael Cogswell Meng Ye Karan Sikka Abhinav Shrivastava Ajay Divakaran 3DV 93 1 1 30 Nov 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Shafiq Joty Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 151 61 0 30 Nov 2023
ViT-Lens: Towards Omni-modal Representations Weixian Lei Yixiao Ge Kun Yi Jianfeng Zhang Difei Gao Dylan Sun Yuying Ge Ying Shan Mike Zheng Shou 99 20 0 27 Nov 2023
Zero-shot audio captioning with audio-language model guidance and audio context keywords Leonard Salewski Stefan Fauth A. Sophia Koepke Zeynep Akata 54 11 0 14 Nov 2023
OmniVec: Learning robust representations with cross modal sharing Siddharth Srivastava Gaurav Sharma SSL 95 67 0 07 Nov 2023
CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition K. A. Noriy Xiaosong Yang Marcin Budka Jian Jun Zhang VLM 79 3 0 18 Oct 2023
Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook Ming Jin Qingsong Wen Yuxuan Liang Chaoli Zhang Siqiao Xue ... Shirui Pan Vincent S. Tseng Yu Zheng Lei Chen Hui Xiong AI4TS SyDa 168 124 0 16 Oct 2023
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale Nina Shvetsova Anna Kukleva Xudong Hong Christian Rupprecht Bernt Schiele Hilde Kuehne 105 26 0 07 Oct 2023
Expedited Training of Visual Conditioned Language Generation via Redundancy Reduction Yiren Jian Tingkai Liu Yunzhe Tao Chunhui Zhang Soroush Vosoughi HX Yang VLM 87 12 0 05 Oct 2023
Human-centric Behavior Description in Videos: New Benchmark and Model Lingru Zhou Yi-Meng Gao Manqing Zhang Peng Wu Peng Wang Yanning Zhang 56 1 0 04 Oct 2023
LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment Bin Zhu Bin Lin Munan Ning Yang Yan Jiaxi Cui ... Zongwei Li Wancai Zhang Zhifeng Li Wei Liu Liejie Yuan VLM MLLM 185 229 0 03 Oct 2023
InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation Yi Wang Yinan He Yizhuo Li Kunchang Li Jiashuo Yu ... Ping Luo Ziwei Liu Yali Wang Limin Wang Yu Qiao VLM VGen 127 275 0 13 Jul 2023