v1v2v3v4 (latest)

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023

5 June 2023

Hang Zhang

Xin Li

Lidong Bing

MLLM

ArXiv (abs)PDF HTML HuggingFace (19 upvotes)

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 875 papers shown

Title
Vid-Morp: Video Moment Retrieval Pretraining from Unlabeled Videos in the Wild Peijun Bao Chenqi Kong Zihao Shao Boon Poh Ng Meng Hwa Er Alex C. Kot 210 3 0 01 Dec 2024
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 498 5 0 01 Dec 2024
ATP-LLaVA: Adaptive Token Pruning for Large Vision Language ModelsComputer Vision and Pattern Recognition (CVPR), 2024 Xubing Ye Yukang Gan Yixiao Ge Xiao Zhang Yansong Tang 287 28 0 30 Nov 2024
Circumventing shortcuts in audio-visual deepfake detection datasets with unsupervised learningComputer Vision and Pattern Recognition (CVPR), 2024 Dragos-Alexandru Boldisor Stefan Smeu Dan Oneaţă Elisabeta Oneata 582 7 0 29 Nov 2024
TimeMarker: A Versatile Video-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability Shimin Chen Xiaohan Lan Yitian Yuan Zequn Jie Lin Ma VLM MLLM 256 38 0 27 Nov 2024
VideoOrion: Tokenizing Object Dynamics in Videos Yicheng Feng Yijiang Li Wanpeng Zhang Sipeng Zheng Zongqing Lu Sipeng Zheng Zongqing Lu 330 7 0 25 Nov 2024
Leveraging the Power of MLLMs for Gloss-Free Sign Language Translation Jungeun Kim Hyeongwoo Jeon Jongseong Bae Ha Young Kim SLR 268 3 0 25 Nov 2024
freePruner: A Training-free Approach for Large Multimodal Model Acceleration Bingxin Xu Yuzhang Shang Yunhao Ge Qian Lou Yan Yan 238 5 0 23 Nov 2024
ReWind: Understanding Long Videos with Instructed Learnable MemoryComputer Vision and Pattern Recognition (CVPR), 2024 Anxhelo Diko Tinghuai Wang Wassim Swaileh Shiyan Sun Ioannis Patras KELM VLM 299 4 0 23 Nov 2024
ReVisionLLM: Recursive Vision-Language Model for Temporal Grounding in Hour-Long VideosComputer Vision and Pattern Recognition (CVPR), 2024 Tanveer Hannan Md. Mohaiminul Islam Jindong Gu Thomas Seidl Gedas Bertasius VLM 150 8 0 22 Nov 2024
VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame SelectionComputer Vision and Pattern Recognition (CVPR), 2024 Songhao Han Wei Huang Hairong Shi Le Zhuo Xiu Su Shifeng Zhang Xu Zhou Xiaojuan Qi Yue Liao Si Liu VGen LRM 230 44 0 22 Nov 2024
Neuro-Symbolic Evaluation of Text-to-Video Models using Formal VerificationComputer Vision and Pattern Recognition (CVPR), 2024 S P Sharan Minkyu Choi Sahil Shah Harsh Goel Mohammad Omama Sandeep Chinchali EGVM 507 5 0 22 Nov 2024
Beyond Training: Dynamic Token Merging for Zero-Shot Video Understanding Yiming Zhang Zhuokai Zhao Zhaorun Chen Zenghui Ding Xianjun Yang Yining Sun 993 8 0 21 Nov 2024
On the Consistency of Video Large Language Models in Temporal ComprehensionComputer Vision and Pattern Recognition (CVPR), 2024 Minjoon Jung Junbin Xiao Byoung-Tak Zhang Angela Yao 390 5 0 20 Nov 2024
Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehension Yongdong Luo Xiawu Zheng Guilin Li Guilin Li Haojia Lin ... Jinfa Huang Jiayi Ji Jiebo Luo Rongrong Ji Rongrong Ji VLM 526 65 0 20 Nov 2024
Generative Timelines for Instructed Visual Assembly Alejandro Pardo Jui-hsien Wang Guohao Li Josef Sivic Bryan C. Russell Fabian Caba Heilbron VGen 195 0 0 19 Nov 2024
On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation Can Cui Zichong Yang Yupeng Zhou Juntong Peng Sung-Yeon Park ... Yiheng Feng Jitesh Panchal Lingxi Li Yaobin Chen Ziran Wang 210 15 0 17 Nov 2024
TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models Tingyu Qu Mingxiao Li Tinne Tuytelaars Marie-Francine Moens VLM 224 3 0 17 Nov 2024
DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization C. Koutlis Symeon Papadopoulos 362 7 0 15 Nov 2024
Jailbreak Attacks and Defenses against Multimodal Generative Models: A Survey Xuannan Liu Xing Cui Peipei Li Zekun Li Huaibo Huang Shuhan Xia Miaoxuan Zhang Yueying Zou Ran He AAML 370 21 0 14 Nov 2024
VideoCogQA: A Controllable Benchmark for Evaluating Cognitive Abilities in Video-Language Models Chenglin Li Qianglong Chen Zhi Li Feng Tao Yin Zhang 342 0 0 14 Nov 2024
Spider: Any-to-Many Multimodal LLM Jinxiang Lai Jie Zhang Jun Liu Jian Li Xiaocheng Lu Song Guo MLLM 436 4 0 14 Nov 2024
Multimodal Instruction Tuning with Hybrid State Space Models Jianing Zhou Han Li Shuai Zhang Ning Xie Ruijie Wang Xiaohan Nie Sheng Liu Lingyun Wang 205 0 0 13 Nov 2024
Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language ModelsComputer Vision and Pattern Recognition (CVPR), 2024 Quan Zhang Yuxin Qi Rui Yuan Xi Tang Yuxin Qi Ke Zhang Chun Yuan 210 5 0 13 Nov 2024
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook Meng Yang Tianqing Zhu Chi Liu Wanlei Zhou Shui Yu Philip S. Yu AAML ELM PILM 234 2 0 12 Nov 2024
HourVideo: 1-Hour Video-Language UnderstandingNeural Information Processing Systems (NeurIPS), 2024 Keshigeyan Chandrasegaran Agrim Gupta Lea M. Hadzic Taran Kota Jimming He Cristobal Eyzaguirre Zane Durante Pengfei Yu Jiajun Wu L. Fei-Fei VLM 213 82 0 07 Nov 2024
TAP-VL: Text Layout-Aware Pre-training for Enriched Vision-Language Models Jonathan Fhima Elad Ben Avraham Oren Nuriel Yair Kittenplon Roy Ganz Aviad Aberdam Ron Litman VLM 200 1 0 07 Nov 2024
VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in VideosComputer Vision and Pattern Recognition (CVPR), 2024 Shehan Munasinghe Hanan Gani Wenqi Zhu Jiale Cao Eric P. Xing Fahad Shahbaz Khan Salman Khan MLLM VGen VLM 367 28 0 07 Nov 2024
MME-Finance: A Multimodal Finance Benchmark for Expert-level Understanding and Reasoning Ziliang Gan Shilin Zhou D. Zhang Yu Lu Che Liu ... Haipang Wu Chaoyou Fu Z. Xu Rongjunchen Zhang Yong Dai 213 26 0 05 Nov 2024
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios Yunkai Dang Mengxi Gao Yibo Yan Xin Zou Yanggan Gu ... Jingyu Wang Peijie Jiang Aiwei Liu Jia Liu Xuming Hu 271 10 0 05 Nov 2024
PPLLaVA: Varied Video Sequence Understanding With Prompt Guidance Ruyang Liu Haoran Tang Haibo Liu Yixiao Ge Mingyu Ding Chen Li Jiankun Yang VLM 170 16 0 04 Nov 2024
Contrasting with Symile: Simple Model-Agnostic Representation Learning for Unlimited ModalitiesNeural Information Processing Systems (NeurIPS), 2024 A. Saporta A. Puli Mark Goldstein Rajesh Ranganath SSL 219 8 0 01 Nov 2024
Generative Emotion Cause Explanation in Multimodal ConversationsInternational Conference on Multimedia Retrieval (ICMR), 2024 Lin Wang Xiaocui Yang Shi Feng Daling Wang Yifei Zhang Zhitao Zhang 390 1 0 01 Nov 2024
On Learning Multi-Modal Forgery Representation for Diffusion Generated Video DetectionNeural Information Processing Systems (NeurIPS), 2024 Xiufeng Song Xiao Guo Junxuan Zhang Qirui Li Lei Bai Xiaoming Liu Guangtao Zhai Xiaohong Liu VGen DiffM 512 26 0 31 Oct 2024
SpeechQE: Estimating the Quality of Direct Speech TranslationConference on Empirical Methods in Natural Language Processing (EMNLP), 2024 HyoJung Han Kevin Duh Marine Carpuat 211 4 0 28 Oct 2024
CT2C-QA: Multimodal Question Answering over Chinese Text, Table and ChartACM Multimedia (MM), 2024 Bowen Zhao Tianhao Cheng Yuejie Zhang Ying Cheng Rui Feng Xiaobo Zhang LMTD 129 4 0 28 Oct 2024
Sensor2Text: Enabling Natural Language Interactions for Daily Activity Tracking Using Wearable SensorsProceedings of the ACM on Interactive Mobile Wearable and Ubiquitous Technologies (IMWUT), 2024 Wenqiang Chen Jiaxuan Cheng Leyao Wang Wei Zhao Wojciech Matusik 228 9 0 26 Oct 2024
GiVE: Guiding Visual Encoder to Perceive Overlooked Information Junjie Li Jianghong Ma Xiaofeng Zhang Yuhang Li Jianyang Shi 287 1 0 26 Oct 2024
FLAASH: Flow-Attention Adaptive Semantic Hierarchical Fusion for Multi-Modal Tobacco Content Analysis N. V. R. Chappa P. Dobbs Bhiksha Raj Khoa Luu 242 3 0 25 Oct 2024
TimeSuite: Improving MLLMs for Long Video Understanding via Grounded TuningInternational Conference on Learning Representations (ICLR), 2024 Xiangyu Zeng Kunchang Li Chenting Wang Xinhao Li Tianxiang Jiang ... Zhengrong Yue Yi Wang Yali Wang Yu Qiao Limin Wang MLLM VLM AI4TS 223 50 0 25 Oct 2024
MMAU: A Massive Multi-Task Audio Understanding and Reasoning BenchmarkInternational Conference on Learning Representations (ICLR), 2024 S. Sakshi Utkarsh Tyagi Sonal Kumar Ashish Seth Ramaneswaran Selvakumar Oriol Nieto R. Duraiswami Sreyan Ghosh Dinesh Manocha AuLLM ELM 204 138 0 24 Oct 2024
AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language ModelsInternational Conference on Learning Representations (ICLR), 2024 Kim Sung-Bin Oh Hyun-Bin JungMok Lee Arda Senocak Joon Son Chung Tae-Hyun Oh MLLM VLM 358 14 0 23 Oct 2024
Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models Zhijie Tan Xu Chu Weiping Li Tong Mo 152 6 0 22 Oct 2024
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Bin Wang Jiaqi Wang Feng Wu Dahua Lin VLM 259 122 0 22 Oct 2024
Mitigating Object Hallucination via Concentric Causal AttentionNeural Information Processing Systems (NeurIPS), 2024 Yun Xing Yiheng Li Ivan Laptev Shijian Lu 190 37 0 21 Oct 2024
OpenMU: Your Swiss Army Knife for Music Understanding Mengjie Zhao Zhi-Wei Zhong Zhuoyuan Mao Shiqi Yang Wei-Hsiang Liao Shusuke Takahashi Hiromi Wakaki Yuki Mitsufuji OSLM 249 10 0 21 Oct 2024
xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs Michael S Ryoo Honglu Zhou Shrikant B. Kendre Can Qin Le Xue ... Kanchana Ranasinghe Caiming Xiong Ran Xu Caiming Xiong Juan Carlos Niebles VGen 242 25 0 21 Oct 2024
Assistive AI for Augmenting Human Decision-making Natabara Máté Gyöngyössy Bernát Török Csilla Farkas Laura Lucaj Attila Menyhárd Krisztina Menyhárd-Balázs András Simonyi Patrick van der Smagt Zsolt Ződi András Lőrincz 241 0 0 18 Oct 2024
Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models Olga Loginova Oleksandr Bezrukov Ravi Shekhar Alexey Kravets 201 3 0 18 Oct 2024
MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations Liang Xu Shaoyang Hua Zili Lin Yifan Liu Feipeng Ma Yichao Yan Xin Jin Xiaokang Yang Wenjun Zeng VGen 212 13 0 17 Oct 2024