Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models

8 June 2023

Salman Khan

Papers citing "Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models"

50 / 462 papers shown

Title
Navigation Instruction Generation with BEV Perception and Large Language Models Sheng Fan Rui Liu Wenguan Wang Yi Yang 50 5 0 21 Jul 2024
Prior Knowledge Integration via LLM Encoding and Pseudo Event Regulation for Video Moment Retrieval Yiyang Jiang Wengyu Zhang Xu-Lu Zhang Xiaoyong Wei Chang Wen Chen Qing Li 46 4 0 21 Jul 2024
ViLLa: Video Reasoning Segmentation with Large Language Model Rongkun Zheng Lu Qi Xi Chen Yi Wang Kun Wang Yu Qiao Hengshuang Zhao VOS LRM 80 2 0 18 Jul 2024
Goldfish: Vision-Language Understanding of Arbitrarily Long Videos Kirolos Ataallah Xiaoqian Shen Eslam Abdelrahman Essam Sleiman Mingchen Zhuge Jian Ding Deyao Zhu Jürgen Schmidhuber Mohamed Elhoseiny VLM 30 18 0 17 Jul 2024
VISA: Reasoning Video Object Segmentation via Large Language Models Cilin Yan Haochen Wang Shilin Yan Xiaolong Jiang Yao Hu Guoliang Kang Weidi Xie E. Gavves LRM VLM VOS 50 28 0 16 Jul 2024
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models Haodong Duan Junming Yang Junming Yang Xinyu Fang Lin Chen ... Yuhang Zang Pan Zhang Jiaqi Wang Dahua Lin Kai Chen LM&MA VLM 46 119 0 16 Jul 2024
Refusing Safe Prompts for Multi-modal Large Language Models Zedian Shao Hongbin Liu Yuepeng Hu Neil Zhenqiang Gong MLLM LRM 46 1 0 12 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 65 4 0 11 Jul 2024
Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding Minghui Wu Chenxu Zhao Anyang Su Donglin Di Tianyu Fu ... Min He Ya Gao Meng Ma Kun Yan Ping Wang 35 0 0 11 Jul 2024
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models Feng Li Renrui Zhang Hao Zhang Yuanhan Zhang Bo Li Wei Li Zejun Ma Chunyuan Li MLLM VLM 52 197 0 10 Jul 2024
AffectGPT: Dataset and Framework for Explainable Multimodal Emotion Recognition Zheng Lian Haiyang Sun Guoying Zhao Jiangyan Yi Bin Liu Jianhua Tao 60 2 0 10 Jul 2024
Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision Orr Zohar Xiaohan Wang Yonatan Bitton Idan Szpektor Serena Yeung-Levy VLM LRM 63 8 0 08 Jul 2024
Multimodal Language Models for Domain-Specific Procedural Video Summarization Nafisa Hussain 50 0 0 07 Jul 2024
OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding Tiancheng Zhao Qianqian Zhang Kyusong Lee Peng Liu Lu Zhang Chunxin Fang Jiajia Liao Kelei Jiang Yibo Ma Ruochen Xu MLLM VLM 59 5 0 06 Jul 2024
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output Pan Zhang Xiaoyi Dong Yuhang Zang Yuhang Cao Rui Qian ... Kai Chen Jifeng Dai Yu Qiao Dahua Lin Jiaqi Wang 47 100 0 03 Jul 2024
Video Watermarking: Safeguarding Your Video from (Unauthorized) Annotations by Video-based LLMs Jinmin Li Kuofeng Gao Yang Bai Jingyun Zhang Shu-Tao Xia 53 4 0 02 Jul 2024
Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time Sanjoy Chowdhury Sayan Nag Subhrajyoti Dasgupta Jun Chen Mohamed Elhoseiny Ruohan Gao Dinesh Manocha VLM MLLM 49 9 0 01 Jul 2024
Tarsier: Recipes for Training and Evaluating Large Video Description Models Jiawei Wang Liping Yuan Yuchen Zhang 49 52 0 30 Jun 2024
Hierarchical Memory for Long Video QA Yiqin Wang Haoji Zhang Yansong Tang Yong-Jin Liu Jiashi Feng Jifeng Dai Xiaojie Jin 75 2 0 30 Jun 2024
ReXTime: A Benchmark Suite for Reasoning-Across-Time in Videos Jr-Jen Chen Yu-Chien Liao Hsi-Che Lin Yu-Chu Yu Yen-Chun Chen Yu-Chiang Frank Wang 37 10 0 27 Jun 2024
MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning Xiangyu Zhao Xiangtai Li Haodong Duan Haian Huang Yining Li Kai Chen Hua Yang VLM MLLM 50 10 0 25 Jun 2024
Less can be more: representational vs. stereotypical gender bias in facial expression recognition Iris Dominguez-Catena D. Paternain A. Jurio M. Galar 42 4 0 25 Jun 2024
Zero-Shot Long-Form Video Understanding through Screenplay Yongliang Wu Bozheng Li Jiawang Cao Wenbo Zhu Yi Lu ... Chuyun Xie Haolin Zheng Ziyue Su Jay Wu Xu Yang 48 4 0 25 Jun 2024
Long Context Transfer from Language to Vision Peiyuan Zhang Kaichen Zhang Bo Li Guangtao Zeng Jingkang Yang Yuanhan Zhang Ziyue Wang Haoran Tan Chunyuan Li Ziwei Liu VLM 72 146 0 24 Jun 2024
Directed Domain Fine-Tuning: Tailoring Separate Modalities for Specific Training Tasks Daniel Wen Nafisa Hussain 107 0 0 24 Jun 2024
VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models Yuxuan Wang Yueqian Wang Dongyan Zhao Cihang Xie Zilong Zheng MLLM VLM 59 26 0 24 Jun 2024
video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models Guangzhi Sun Wenyi Yu Changli Tang Xianzhao Chen Tian Tan Wei Li Lu Lu Zejun Ma Yuxuan Wang Chao Zhang 47 23 0 22 Jun 2024
MMBench-Video: A Long-Form Multi-Shot Benchmark for Holistic Video Understanding Xinyu Fang Kangrui Mao Haodong Duan Xiangyu Zhao Yining Li Dahua Lin Kai Chen VLM 57 63 0 20 Jun 2024
Through the Theory of Mind's Eye: Reading Minds with Multimodal Video Large Language Models Zhawnen Chen Tianchun Wang Yizhou Wang Michal Kosinski Xiang Zhang Yun Fu Sheng Li LRM 34 2 0 19 Jun 2024
VoCo-LLaMA: Towards Vision Compression with Large Language Models Xubing Ye Yukang Gan Xiaoke Huang Yixiao Ge Yansong Tang MLLM VLM 45 23 0 18 Jun 2024
LLaNA: Large Language and NeRF Assistant Andrea Amaduzzi Pierluigi Zama Ramirez Giuseppe Lisanti Samuele Salti Luigi Di Stefano 53 2 0 17 Jun 2024
MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs Ziyu Liu Tao Chu Yuhang Zang Xilin Wei Xiaoyi Dong ... Zijian Liang Yuanjun Xiong Yu Qiao Dahua Lin Jiaqi Wang VLM 45 35 0 17 Jun 2024
VideoLLM-online: Online Video Large Language Model for Streaming Video Joya Chen Zhaoyang Lv Shiwei Wu Kevin Qinghong Lin Chenan Song Difei Gao Jia-Wei Liu Ziteng Gao Dongxing Mao Mike Zheng Shou MLLM MoMe 57 50 0 17 Jun 2024
VideoVista: A Versatile Benchmark for Video Understanding and Reasoning Yunxin Li Xinyu Chen Baotian Hu Longyue Wang Haoyuan Shi Min-Ling Zhang MLLM LRM 63 26 0 17 Jun 2024
Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning Zebang Cheng Zhi-Qi Cheng Jun-Yan He Jingdong Sun Kai Wang Yuxiang Lin Zheng Lian Xiaojiang Peng Alexander G. Hauptmann MLLM 40 31 0 17 Jun 2024
Beyond Raw Videos: Understanding Edited Videos with Large Multimodal Model Lu Xu Sijie Zhu Chunyuan Li Chia-Wen Kuo Fan Chen Xinyao Wang Guang Chen Dawei Du Ye Yuan Longyin Wen 44 4 0 15 Jun 2024
GPT-4o: Visual perception performance of multimodal large language models in piglet activity understanding Yiqi Wu Xiaodan Hu Ziming Fu Siling Zhou Jiangong Li MLLM 40 10 0 14 Jun 2024
A Survey of Video Datasets for Grounded Event Understanding Kate Sanders Benjamin Van Durme 47 4 0 14 Jun 2024
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs Rohit K Bharadwaj Hanan Gani Muzammal Naseer Fahad Shahbaz Khan Salman Khan 70 3 0 14 Jun 2024
VideoGPT+: Integrating Image and Video Encoders for Enhanced Video Understanding Muhammad Maaz H. Rasheed Salman Khan Fahad A Khan VLM MLLM 42 51 0 13 Jun 2024
Explore the Limits of Omni-modal Pretraining at Scale Yiyuan Zhang Handong Li Jing Liu Xiangyu Yue VLM LRM 49 1 0 13 Jun 2024
MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding Fei Wang Xingyu Fu James Y. Huang Zekun Li Qin Liu ... Kai-Wei Chang Dan Roth Sheng Zhang Hoifung Poon Muhao Chen VLM 50 47 0 13 Jun 2024
Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs Zijia Zhao Haoyu Lu Yuqi Huo Yifan Du Tongtian Yue Longteng Guo Bingning Wang Weipeng Chen Jing Liu 49 2 0 13 Jun 2024
Too Many Frames, Not All Useful: Efficient Strategies for Long-Form Video QA Jongwoo Park Kanchana Ranasinghe Kumara Kahatapitiya Wonjeong Ryoo Donghyun Kim Michael S. Ryoo 65 20 0 13 Jun 2024
MMWorld: Towards Multi-discipline Multi-faceted World Model Evaluation in Videos Xuehai He Weixi Feng Kaizhi Zheng Yujie Lu Wanrong Zhu ... Zhengyuan Yang Kevin Lin William Yang Wang Lijuan Wang Xin Eric Wang VGen LRM 51 12 0 12 Jun 2024
Flash-VStream: Memory-Based Real-Time Understanding for Long Video Streams Haoji Zhang Yiqin Wang Yansong Tang Yong-Jin Liu Jiashi Feng Jifeng Dai Xiaojie Jin 52 38 0 12 Jun 2024
LVBench: An Extreme Long Video Understanding Benchmark Weihan Wang Zehai He Wenyi Hong Yean Cheng Xiaohan Zhang ... Shiyu Huang Bin Xu Yuxiao Dong Ming Ding Jie Tang ELM VLM 49 67 0 12 Jun 2024
Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models Shimin Chen Yitian Yuan Shaoxiang Chen Zequn Jie Lin Ma VLM 35 3 0 12 Jun 2024
TRINS: Towards Multimodal Language Models that Can Read Ruiyi Zhang Yanzhe Zhang Jian Chen Yufan Zhou Jiuxiang Gu Changyou Chen Tong Sun VLM 39 6 0 10 Jun 2024
iMotion-LLM: Motion Prediction Instruction Tuning Abdulwahab Felemban Eslam Mohamed Bakr Xiaoqian Shen Jian Ding Abduallah A. Mohamed Mohamed Elhoseiny 60 1 0 10 Jun 2024