Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 702 papers shown

Title
AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information? Kaixiong Gong Kaituo Feng Yangqiu Song Yibing Wang Mofan Cheng ... Jiaming Han Benyou Wang Yutong Bai Zheng Yang Xiangyu Yue MLLM AuLLM VLM 91 6 0 03 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 108 1 0 03 Dec 2024
PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos Meng Cao Haoran Tang Haoze Zhao Hangyu Guo Jing Liu Ge Zhang Ruyang Liu Qiang Sun Ian Reid Xiaodan Liang 104 2 0 02 Dec 2024
Eyes on the Road: State-of-the-Art Video Question Answering Models Assessment for Traffic Monitoring Tasks Joseph Raj Vishal Divesh Basina Aarya Choudhary Bharatesh Chakravarthi 67 1 0 02 Dec 2024
SEAL: Semantic Attention Learning for Long Video Representation Lan Wang Yujia Chen Wen-Sheng Chu Vishnu Boddeti Du Tran VLM 83 0 0 02 Dec 2024
Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild Peijun Bao Chenqi Kong Zihao Shao Boon Poh Ng Meng Hwa Er Alex C. Kot 74 2 0 01 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 106 2 0 01 Dec 2024
ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models Xubing Ye Yukang Gan Yixiao Ge Xiao Zhang Yansong Tang 101 7 0 30 Nov 2024
Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learning Dragos-Alexandru Boldisor Stefan Smeu Dan Oneaţă Elisabeta Oneata 109 1 0 29 Nov 2024
TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability Shimin Chen Xiaohan Lan Yitian Yuan Zequn Jie Lin Ma VLM MLLM 87 13 0 27 Nov 2024
MotionLLaMA: A Unified Framework for Motion Synthesis and Comprehension Zeyu Ling Bo Han Shiyang Li H. Shen Jikang Cheng Changqing Zou 89 1 0 26 Nov 2024
Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation Jungeun Kim Hyeongwoo Jeon Jongseong Bae Ha Young Kim SLR 87 0 0 25 Nov 2024
VideoOrion: Tokenizing Object Dynamics in Videos Yicheng Feng Yijiang Li Wanpeng Zhang Sipeng Zheng Zongqing Lu Sipeng Zheng Zongqing Lu 109 1 0 25 Nov 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 99 3 0 23 Nov 2024
ReWind: Understanding Long Videos with Instructed Learnable Memory Anxhelo Diko Tinghuai Wang Wassim Swaileh Shiyan Sun Ioannis Patras KELM VLM 84 0 0 23 Nov 2024
Neuro-Symbolic Evaluation of Text-to-Video Models using Formal Verification S P Sharan Minkyu Choi Sahil Shah Harsh Goel Mohammad Omama Sandeep Chinchali EGVM 108 3 0 22 Nov 2024
Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding Yiming Zhang Zhuokai Zhao Zhaorun Chen Zenghui Ding Xianjun Yang Yining Sun 278 1 0 21 Nov 2024
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension Yongdong Luo Xiawu Zheng Xiao Yang Guilin Li Haojia Lin Jinfa Huang Jiayi Ji Rongrong Ji Jiebo Luo Rongrong Ji VLM 85 18 0 20 Nov 2024
On the Consistency of Video Large Language Models in Temporal Comprehension Minjoon Jung Junbin Xiao Byoung-Tak Zhang Angela Yao 89 2 0 20 Nov 2024
Generative Timelines for Instructed Visual Assembly Alejandro Pardo Jui-hsien Wang Guohao Li Josef Sivic Bryan C. Russell Fabian Caba Heilbron VGen 72 0 0 19 Nov 2024
On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation Can Cui Zichong Yang Yupeng Zhou Juntong Peng Sung-Yeon Park ... Yiheng Feng Jitesh Panchal Lingxi Li Yaobin Chen Ziran Wang 79 7 0 17 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 47 2 0 17 Nov 2024
DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization C. Koutlis Symeon Papadopoulos 61 2 0 15 Nov 2024
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey Xuannan Liu Xing Cui Peipei Li Zekun Li Huaibo Huang Shuhan Xia Miaoxuan Zhang Yueying Zou Ran He AAML 67 8 0 14 Nov 2024
VCBench: A Controllable Benchmark for Symbolic and Abstract Challenges in Video Cognition Chenglin Li Qianglong Chen Zhi Li Feng Tao Yin Zhang 41 0 0 14 Nov 2024
Spider: Any-to-Many Multimodal LLM Jinxiang Lai Jie Zhang Jun Liu Jian Li Xiaocheng Lu Song Guo MLLM 72 2 0 14 Nov 2024
Multimodal Instruction Tuning with Hybrid State Space Models Jianing Zhou Han Li Shuai Zhang Ning Xie Ruijie Wang Xiaohan Nie Sheng Liu Lingyun Wang 46 0 0 13 Nov 2024
Can MLLMs Guide Weakly-Supervised Temporal Action Localization Tasks? Quan Zhang Yuxin Qi 53 2 0 13 Nov 2024
Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos Sagnik Majumder Tushar Nagarajan Ziad Al-Halah Reina Pradhan Kristen Grauman 41 1 0 13 Nov 2024
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook Meng Yang Tianqing Zhu Chi Liu Wanlei Zhou Shui Yu Philip S. Yu AAML ELM PILM 69 1 0 12 Nov 2024
EVQAScore: A Fine-grained Metric for Video Question Answering Data Quality Evaluation Hao Liang Zirong Chen Feiyu Xiong Wentao Zhang 38 1 0 11 Nov 2024
HourVideo: 1-Hour Video-Language Understanding Keshigeyan Chandrasegaran Agrim Gupta Lea M. Hadzic Taran Kota Jimming He Cristobal Eyzaguirre Zane Durante Manling Li Jiajun Wu L. Fei-Fei VLM 56 35 0 07 Nov 2024
TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models Jonathan Fhima Elad Ben Avraham Oren Nuriel Yair Kittenplon Roy Ganz Aviad Aberdam Ron Litman VLM 42 1 0 07 Nov 2024
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos Shehan Munasinghe Hanan Gani Wenqi Zhu Jiale Cao Eric P. Xing Fahad Shahbaz Khan Salman Khan MLLM VGen VLM 44 6 0 07 Nov 2024
MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning Ziliang Gan Yu Lu D. Zhang Haohan Li Che Liu ... Haipang Wu Chaoyou Fu Z. Xu Rongjunchen Zhang Yong Dai 57 4 0 05 Nov 2024
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance Ruyang Liu Haoran Tang Haibo Liu Yixiao Ge Ying Shan Chen Li Jiankun Yang VLM 53 6 0 04 Nov 2024
Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited Modalities A. Saporta A. Puli Mark Goldstein Rajesh Ranganath SSL 50 0 0 01 Nov 2024
Generative Emotion Cause Explanation in Multimodal Conversations Lin Wang Xiaocui Yang Shi Feng Daling Wang Yifei Zhang 39 0 0 01 Nov 2024
On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detection Xiufeng Song Xiao Guo J. Zhang Qirui Li Lei Bai Xiaoming Liu Guangtao Zhai Xiaohong Liu DiffM VGen 74 9 0 31 Oct 2024
SpeechQE: Estimating the Quality of Direct Speech Translation HyoJung Han Kevin Duh Marine Carpuat 34 0 0 28 Oct 2024
CT2C-QA: Multimodal Question Answering over Chinese Text, Table and Chart Bowen Zhao Tianhao Cheng Yuejie Zhang Ying Cheng Rui Feng Xiaobo Zhang LMTD 37 1 0 28 Oct 2024
Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable Sensors Wenqiang Chen Jiaxuan Cheng Leyao Wang Wei Zhao Wojciech Matusik 33 1 0 26 Oct 2024
GiVE: Guiding Visual Encoder to Perceive Overlooked Information Junjie Li Jianghong Ma Xiaofeng Zhang Yuhang Li Jianyang Shi 48 0 0 26 Oct 2024
FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysis N. V. R. Chappa P. Dobbs Bhiksha Raj Khoa Luu 42 3 0 25 Oct 2024
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded Tuning Xiangyu Zeng Kunchang Li Chenting Wang Xinhao Li Tianxiang Jiang ... Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang MLLM VLM AI4TS 71 15 0 25 Oct 2024
MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark S. Sakshi Utkarsh Tyagi Sonal Kumar Ashish Seth Ramaneswaran Selvakumar Oriol Nieto R. Duraiswami Sreyan Ghosh Dinesh Manocha AuLLM ELM 77 28 0 24 Oct 2024
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models Kim Sung-Bin Oh Hyun-Bin JungMok Lee Arda Senocak Joon Son Chung Tae-Hyun Oh MLLM VLM 50 3 0 23 Oct 2024
Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models Zhijie Tan Xu Chu Weiping Li Tong Mo 33 1 0 22 Oct 2024
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Zeang Sheng Jiaqi Wang Feng Wu Dahua Lin VLM 53 29 0 22 Oct 2024
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs Michael S Ryoo Honglu Zhou Shrikant B. Kendre Can Qin Le Xue Manli Shu Silvio Savarese Ran Xu Caiming Xiong Juan Carlos Niebles VGen 50 13 0 21 Oct 2024