v1v2v3v4 (latest)

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

5 June 2023

Hang Zhang

Xin Li

Lidong Bing

MLLM

ArXiv (abs)PDF HTML HuggingFace (19 upvotes)

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 875 papers shown

Title
HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding Jiaxing Zhao Q. Yang Yixing Peng Detao Bai Shimin Yao ... Xiang Chen Shenghao Fu Weixuan chen Xihan Wei Liefeng Bo VGen AuLLM 207 27 0 28 Jan 2025
ENTER: Event Based Interpretable Reasoning for VideoQA Hammad A. Ayyubi Junzhang Liu Ali Asgarov Zaber Ibn Abdul Hakim Najibul Haque Sarker ... Md. Atabuzzaman Xudong Lin Naveen Reddy Dyava Shih-Fu Chang Chris Thomas NAI 466 3 0 24 Jan 2025
MMVU: Measuring Expert-Level Multi-Discipline Video UnderstandingComputer Vision and Pattern Recognition (CVPR), 2025 Yilun Zhao Lujing Xie Haowei Zhang Guo Gan Yitao Long ... Xiangru Tang Zhenwen Liang Yongxu Liu Chen Zhao Arman Cohan 259 59 0 21 Jan 2025
Omni-RGPT: Unifying Image and Video Region-level Understanding via Token MarksComputer Vision and Pattern Recognition (CVPR), 2025 Miran Heo Min-Hung Chen De-An Huang Sifei Liu Subhashree Radhakrishnan Seon Joo Kim Yu-Chun Wang Ryo Hachiuma ObjD VLM 468 6 0 14 Jan 2025
Initial Findings on Sensor based Open Vocabulary Activity Recognition via Text Embedding Inversion L. Ray Bo Zhou Sungho Suh P. Lukowicz VLM 237 0 0 13 Jan 2025
TimeLogic: A Temporal Logic Benchmark for Video QA S. Swetha Hilde Kuehne Mubarak Shah 113 7 0 13 Jan 2025
VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video CaptioningAAAI Conference on Artificial Intelligence (AAAI), 2025 Ji Soo Lee Jongha Kim Jeehye Na Jinyoung Park H. Kim VGen 103 7 0 12 Jan 2025
OneLLM: One Framework to Align All Modalities with LanguageComputer Vision and Pattern Recognition (CVPR), 2023 Jiaming Han Kaixiong Gong Yiyuan Zhang Yuan Liu Kaipeng Zhang Dahua Lin Yu Qiao Shiyang Feng Xiangyu Yue MLLM 484 188 0 10 Jan 2025
H-MBA: Hierarchical MamBa Adaptation for Multi-Modal Video Understanding in Autonomous DrivingAAAI Conference on Artificial Intelligence (AAAI), 2025 Tian Jin Yuxiao Luo Yue Ma Yu Qiao Yali Wang Mamba 222 5 0 08 Jan 2025
Video-of-Thought: Step-by-Step Video Reasoning from Perception to CognitionInternational Conference on Machine Learning (ICML), 2024 Hao Fei Shengqiong Wu Wei Ji Hao Zhang Hao Fei Yang Deng Wynne Hsu LRM VGen 349 140 0 08 Jan 2025
LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision TokenInternational Conference on Learning Representations (ICLR), 2025 Shaolei Zhang Qingkai Fang Zhe Yang Yang Feng MLLM VLM 356 91 0 07 Jan 2025
Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos Haobo Yuan Xianrui Li Tao Zhang Zilong Huang Shilin Xu ... Yunhai Tong Lu Qi Jiashi Feng Ming-Hsuan Yang Ming-Hsuan Yang VLM 494 68 0 07 Jan 2025
MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language ModelsComputer Vision and Pattern Recognition (CVPR), 2025 Wenyi Hong Yean Cheng Zhiyong Yang Weihan Wang Lefan Wang Xiaotao Gu Xiaotao Gu Yuxiao Dong J. Tang CoGe VLM 213 21 0 06 Jan 2025
Visual Large Language Models for Generalized and Specialized Applications Jiayi Zhang Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 390 32 0 06 Jan 2025
FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance Haicheng Wang Zhemeng Yu Gabriele Spadaro Chen Ju Victor Quétu Enzo Tartaglione Enzo Tartaglione VLM 858 14 0 05 Jan 2025
Listening and Seeing Again: Generative Error Correction for Audio-Visual Speech RecognitionInformation Fusion (Inf. Fusion), 2025 Rui Liu Hongyu Yuan Hong Li 204 2 0 03 Jan 2025
OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language ModelsIEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024 L. Ray Bo Zhou Sungho Suh P. Lukowicz VLM 87 0 0 03 Jan 2025
Multimodal Large Models Are Effective Action AnticipatorsIEEE transactions on multimedia (TMM), 2025 Binglu Wang Yao Tian Shunzhou Wang Le Yang OffRL 105 5 0 03 Jan 2025
GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models Zhangyang Qi Zhixiong Zhang Ye Fang Yuan Liu Hengshuang Zhao 571 47 0 02 Jan 2025
Transferability of Adversarial Attacks in Video-based MLLMs: A Cross-modal Image-to-Video Approach Linhao Huang Xue Jiang Zhiqiang Wang Wentao Mo Xi Xiao Bo Han Yongjie Yin Feng Zheng AAML 517 6 0 02 Jan 2025
2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining Wenqi Zhang Hang Zhang Xin Li Jiashuo Sun Yongliang Shen Weiming Lu Deli Zhao Yueting Zhuang Lidong Bing VLM 519 5 0 01 Jan 2025
Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, EditingNeural Information Processing Systems (NeurIPS), 2024 Hao Fei Shengqiong Wu Hao Zhang Tat-Seng Chua Shuicheng Yan 391 70 0 31 Dec 2024
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLMComputer Vision and Pattern Recognition (CVPR), 2024 Yuqian Yuan Hang Zhang Wentong Li Zesen Cheng Boqiang Zhang ... Deli Zhao Wenqiao Zhang Yueting Zhuang Jianke Zhu Lidong Bing 336 29 0 31 Dec 2024
VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling Xinhao Li Yi Wang Jiashuo Yu Xiangyu Zeng Yuhan Zhu ... Yinan He Chenting Wang Yu Qiao Yali Wang L. Wang VLM 718 95 0 31 Dec 2024
ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding Xiao Wang Qingyi Si Yue Yu Shiyu Zhu Zheng Lin Liqiang Nie VLM 503 23 0 29 Dec 2024
When SAM2 Meets Video Shadow and Mirror Detection Leiping Jie VLM 143 1 0 26 Dec 2024
AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues Se Jin Park Yeonju Kim Hyeongseop Rha Bella Godiva Y. Ro 120 2 0 23 Dec 2024
VidCtx: Context-aware Video Question Answering with Image Models Andreas Goulas Vasileios Mezaris Ioannis Patras 892 2 0 23 Dec 2024
G-VEval: A Versatile Metric for Evaluating Image and Video Captions Using GPT-4oAAAI Conference on Artificial Intelligence (AAAI), 2024 Tony Cheng Tong Sirui He Z. Shao Dit-Yan Yeung 208 15 0 18 Dec 2024
Do Language Models Understand Time?The Web Conference (WWW), 2024 Xi Ding Lei Wang 628 9 0 18 Dec 2024
LLMs are Also Effective Embedding Models: An In-depth Overview Chongyang Tao Tao Shen Shen Gao Junshuo Zhang Zhen Li Kai Hua Wenpeng Hu Zhengwei Tao Shuai Ma 283 27 0 17 Dec 2024
Combating Multimodal LLM Hallucination via Bottom-Up Holistic ReasoningAAAI Conference on Artificial Intelligence (AAAI), 2024 Shengqiong Wu Hao Fei Liangming Pan William Yang Wang Shuicheng Yan Tat-Seng Chua LRM 320 14 0 15 Dec 2024
AgentPS: Agentic Process Supervision for Content Moderation with Multimodal LLMs Gorden Liu Yu Sun R.-H. Sun Xin Dong Hongyu Xiong Hongyu Xiong LLMAG 190 1 0 15 Dec 2024
Empowering LLMs to Understand and Generate Complex Vector GraphicsComputer Vision and Pattern Recognition (CVPR), 2024 Ximing Xing Juncheng Hu Guotao Liang Jing Zhang Dong Xu Qian Yu 390 26 0 15 Dec 2024
PVC: Progressive Visual Token Compression for Unified Image and Video Processing in Large Vision-Language ModelsComputer Vision and Pattern Recognition (CVPR), 2024 Chenyu Yang Xuan Dong X. Zhu Weijie Su Jiahao Wang H. Tian Zheyu Chen Wenhai Wang Lewei Lu Jifeng Dai VLM 172 7 0 12 Dec 2024
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions Pan Zhang Xiaoyi Dong Yuhang Cao Yuhang Zang Rui Qian ... Xinsong Zhang Kai Chen Yu Qiao Dahua Lin Jiaqi Wang KELM 330 31 0 12 Dec 2024
Neptune: The Long Orbit to Benchmarking Long Video Understanding Arsha Nagrani Ruotong Wang Ramin Mehran Rachel Hornung N. B. Gundavarapu ... Boqing Gong Cordelia Schmid Mikhail Sirotenko Yukun Zhu Tobias Weyand 369 14 0 12 Dec 2024
Dynamic-VLM: Simple Dynamic Visual Token Compression for VideoLLM Haobo Wang Yuxiang Nie Yongjie Ye Deng GuanYu Yanjie Wang Shuai Li Haiyang Yu Jinghui Lu Can Huang VLM MLLM 216 13 0 12 Dec 2024
Foundation Models and Adaptive Feature Selection: A Synergistic Approach to Video Question AnsweringIEEE Workshop/Winter Conference on Applications of Computer Vision (WACV), 2024 Sai Bhargav Rongali M. Cui Ankit Jha Neha Bhargava Saurabh Prasad Biplab Banerjee 218 0 0 12 Dec 2024
TimeRefine: Temporal Grounding with Time Refining Video LLM Xizi Wang Feng Cheng Ziyang Wang Huiyu Wang Md. Mohaiminul Islam Lorenzo Torresani Joey Tianyi Zhou Gedas Bertasius David J. Crandall 368 5 0 12 Dec 2024
EgoPlan-Bench2: A Benchmark for Multimodal Large Language Model Planning in Real-World Scenarios Lu Qiu Yuying Ge Yi Chen Yixiao Ge Mingyu Ding Xihui Liu LLMAG LRM 324 18 0 05 Dec 2024
PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation Ao Wang Hui Chen Jianchao Tan Jianchao Tan Xunliang Cai Zijia Lin Jiawei Han Jungong Han Guiguang Ding VLM 396 4 0 04 Dec 2024
Video LLMs for Temporal Reasoning in Long Videos Fawad Javed Fateh Umer Ahmed Hamza Khan M. Zia Quoc-Huy Tran VLM 523 6 0 04 Dec 2024
Medical Multimodal Foundation Models in Clinical Diagnosis and Treatment: Applications, Challenges, and Future DirectionsArtificial Intelligence in Medicine (AIM), 2024 Kai Sun Siyan Xue F. Sun Haoran Sun Yu-Juan Luo ... Xinzhou Wang Lei Yang Shuo Jin Jun Yan Jiahong Dong AI4CE 277 17 0 03 Dec 2024
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? Kaixiong Gong Kaituo Feng Yangqiu Song Yibing Wang Mofan Cheng ... Jiaming Han Benyou Wang Yutong Bai Zhiyong Yang Xiangyu Yue MLLM AuLLM VLM 235 23 0 03 Dec 2024
Progress-Aware Video Frame CaptioningComputer Vision and Pattern Recognition (CVPR), 2024 Zihui Xue Joungbin An Xitong Yang Kristen Grauman 512 5 0 03 Dec 2024
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos Meng Cao Haoran Tang Haoze Zhao Hangyu Guo Jing Liu Ge Zhang Ruyang Liu Qiang Sun Ian Reid Xiaodan Liang 350 9 0 02 Dec 2024
Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks Joseph Raj Vishal Divesh Basina Aarya Choudhary Bharatesh Chakravarthi 312 3 0 02 Dec 2024
SEAL: Semantic Attention Learning for Long Video RepresentationComputer Vision and Pattern Recognition (CVPR), 2024 Lan Wang Yujia Chen Wen-Sheng Chu Vishnu Boddeti Du Tran VLM 410 7 0 02 Dec 2024
VISTA: Enhancing Long-Duration and High-Resolution Video Understanding by Video Spatiotemporal AugmentationComputer Vision and Pattern Recognition (CVPR), 2024 Weiming Ren Huan Yang Jie Min Cong Wei Lei Ma 784 9 0 01 Dec 2024