Audio-Visual LLM for Video Understanding

11 December 2023

Lei Zhang

Papers citing "Audio-Visual LLM for Video Understanding"

37 / 37 papers shown

Title
EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning Zhenghao Xing Xiaowei Hu Chi-Wing Fu W. Wang Jifeng Dai Pheng-Ann Heng MLLM OffRL VLM LRM 50 0 0 07 May 2025
SF2T: Self-supervised Fragment Finetuning of Video-LLMs for Fine-Grained Understanding Yangliu Hu Zikai Song Na Feng Yawei Luo Junqing Yu Yi-Ping Phoebe Chen Wei Yang 33 0 0 10 Apr 2025
Aligned Better, Listen Better for Audio-Visual Large Language Models Yuxin Guo Shuailei Ma Shijie Ma Xiaoyi Bao Chen-Wei Xie Kecheng Zheng Tingyu Weng Siyang Sun Yun Zheng Wei Zou MLLM AuLLM 62 2 0 02 Apr 2025
Scaling Auditory Cognition via Test-Time Compute in Audio Language Models Ting Dang Yan Gao Hong Jia 36 0 0 30 Mar 2025
Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs Sanjoy Chowdhury Hanan Gani Nishit Anand Sayan Nag Ruohan Gao Mohamed Elhoseiny Salman Khan Dinesh Manocha LRM 52 0 0 29 Mar 2025
Video-R1: Reinforcing Video Reasoning in MLLMs Kaituo Feng Kaixiong Gong B. Li Zonghao Guo Yibing Wang Tianshuo Peng J. Wu Xiaoying Zhang Benyou Wang Xiangyu Yue AI4TS SyDa LRM 48 13 0 27 Mar 2025
Being-0: A Humanoid Robotic Agent with Vision-Language Models and Modular Skills Haoqi Yuan Yu Bai Yuhui Fu Bohan Zhou Yicheng Feng Xinrun Xu Yi Zhan Börje F. Karlsson Zongqing Lu LM&Ro 76 0 0 16 Mar 2025
MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality Shuaike Li Kai Zhang Q. Liu Enhong Chen KELM 78 1 0 04 Mar 2025
LASER: Lip Landmark Assisted Speaker Detection for Robustness Le Thien Phuc Nguyen Z. Yu Yong Jae Lee 29 1 0 21 Jan 2025
Do Language Models Understand Time? Xi Ding Lei Wang 178 0 0 18 Dec 2024
Patch-level Sounding Object Tracking for Audio-Visual Question Answering Zhangbin Li Jinxing Zhou J. Zhang Shengeng Tang Kun Li D. Guo 83 4 0 14 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... X. Zhang K. Chen Yu Qiao D. Lin Jiaqi Wang KELM 84 12 0 12 Dec 2024
VideoOrion: Tokenizing Object Dynamics in Videos Yicheng Feng Yijiang Li Wanpeng Zhang Sipeng Zheng Zongqing Lu Sipeng Zheng Zongqing Lu 109 1 0 25 Nov 2024
OMCAT: Omni Context Aware Transformer Arushi Goel Karan Sapra Matthieu Le Rafael Valle Andrew Tao Bryan Catanzaro MLLM VLM 18 0 0 15 Oct 2024
Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs Kai Han Jianyuan Guo Yehui Tang W. He Enhua Wu Yunhe Wang MLLM VLM 21 3 0 14 Oct 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 32 8 0 28 Aug 2024
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time Sanjoy Chowdhury Sayan Nag Subhrajyoti Dasgupta Jun Chen Mohamed Elhoseiny Ruohan Gao Dinesh Manocha VLM MLLM 36 9 0 01 Jul 2024
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models Guangzhi Sun Wenyi Yu Changli Tang Xianzhao Chen Tian Tan Wei Li Lu Lu Zejun Ma Yuxuan Wang Chao Zhang 45 22 0 22 Jun 2024
Joint Embeddings for Graph Instruction Tuning Vlad Argatu Aaron Haag Oliver Lohse 36 0 0 31 May 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping-Chia Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 47 36 0 26 May 2024
V2Xum-LLM: Cross-Modal Video Summarization with Temporal Prompt Instruction Tuning Hang Hua Yunlong Tang Chenliang Xu Jiebo Luo VGen 60 25 0 18 Apr 2024
AVicuna: Audio-Visual LLM with Interleaver and Context-Boundary Alignment for Temporal Referential Dialogue Yunlong Tang Daiki Shimada Jing Bi Chenliang Xu VGen 34 10 0 24 Mar 2024
Hear Me, See Me, Understand Me: Audio-Visual Autism Behavior Recognition Shijian Deng Erin E. Kosloski Siddhi Patel Zeke A. Barnett Yiyang Nan ... William T. Doan Matthew Wang Harsh Singh P. Rollins Yapeng Tian 31 4 0 22 Mar 2024
Reframe Anything: LLM Agent for Open World Video Reframing Jiawang Cao Yongliang Wu Weiheng Chi Wenbo Zhu Ziyue Su Jay Wu 34 3 0 10 Mar 2024
Video Understanding with Large Language Models: A Survey Yunlong Tang Jing Bi Siting Xu Luchuan Song Susan Liang ... Feng Zheng Jianguo Zhang Ping Luo Jiebo Luo Chenliang Xu VLM 54 82 0 29 Dec 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Shafiq R. Joty Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 38 45 0 30 Nov 2023
LLM4TS: Aligning Pre-Trained LLMs as Data-Efficient Time-Series Forecasters Ching Chang Wei-Yao Wang Wenjie Peng Tien-Fu Chen AI4TS 38 45 0 16 Aug 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 270 4,229 0 30 Jan 2023
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,915 0 04 Mar 2022
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection Ke Chen Xingjian Du Bilei Zhu Zejun Ma Taylor Berg-Kirkpatrick Shlomo Dubnov ViT 118 264 0 02 Feb 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,125 0 28 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,434 0 11 Nov 2021
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 259 558 0 28 Sep 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Yin Cui Boqing Gong ViT 248 577 0 22 Apr 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 255 4,777 0 24 Feb 2021
Is Space-Time Attention All You Need for Video Understanding? Gedas Bertasius Heng Wang Lorenzo Torresani ViT 280 1,981 0 09 Feb 2021
ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning Sangho Lee Jiwan Chung Youngjae Yu Gunhee Kim Thomas Breuel Gal Chechik Yale Song 71 45 0 26 Jan 2021