Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

5 June 2023

Xin Li

Papers citing "Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding"

50 / 702 papers shown

Title
Mitigating Object Hallucination via Concentric Causal Attention Yun Xing Yiheng Li Ivan Laptev Shijian Lu 53 19 0 21 Oct 2024
OpenMU: Your Swiss Army Knife for Music Understanding Mengjie Zhao Zhi-Wei Zhong Zhuoyuan Mao Shiqi Yang Wei-Hsiang Liao Shusuke Takahashi Hiromi Wakaki Yuki Mitsufuji OSLM 45 4 0 21 Oct 2024
FIOVA: A Multi-Annotator Benchmark for Human-Aligned Video Captioning Shiyu Hu Xuchen Li Xuzhao Li Jing Zhang Yipei Wang Xin Zhao Kang Hao Cheong VLM 31 1 0 20 Oct 2024
Making Every Frame Matter: Continuous Activity Recognition in Streaming Video via Adaptive Video Context Modeling Hao Wu Donglin Bai Shiqi Jiang Qianxi Zhang Yue Yang Ting Cao Fengyuan Xu Yunxin Liu Fengyuan Xu 190 0 0 19 Oct 2024
Assistive AI for Augmenting Human Decision-making Natabara Máté Gyöngyössy Bernát Török Csilla Farkas Laura Lucaj Attila Menyhárd Krisztina Menyhárd-Balázs András Simonyi Patrick van der Smagt Zsolt Ződi András Lőrincz 41 0 0 18 Oct 2024
Addressing Blind Guessing: Calibration of Selection Bias in Multiple-Choice Question Answering by Video Language Models Olga Loginova Oleksandr Bezrukov Alexey Kravets 28 0 0 18 Oct 2024
MotionBank: A Large-scale Video Motion Benchmark with Disentangled Rule-based Annotations Liang Xu Shaoyang Hua Zili Lin Yifan Liu Feipeng Ma Yichao Yan Xin Jin Xiaokang Yang Wenjun Zeng VGen 41 3 0 17 Oct 2024
Roadmap towards Superhuman Speech Understanding using Large Language Models Fan Bu Yuhao Zhang Xuben Wang Benyou Wang Qiang Liu Yiming Li LM&MA ELM AuLLM 228 1 0 17 Oct 2024
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio Sicong Leng Yun Xing Zesen Cheng Yang Zhou Hang Zhang Xin Li Deli Zhao Shijian Lu Chunyan Miao Lidong Bing 42 8 0 16 Oct 2024
MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs Yunqiu Xu Linchao Zhu Yi Yang 34 3 0 16 Oct 2024
TransAgent: Transfer Vision-Language Foundation Models with Heterogeneous Agent Collaboration Yiwei Guo Shaobin Zhuang Kunchang Li Yu Qiao Yali Wang VLM CLIP 45 0 0 16 Oct 2024
OMCAT: Omni Context Aware Transformer Arushi Goel Karan Sapra Matthieu Le Rafael Valle Andrew Tao Bryan Catanzaro MLLM VLM 36 0 0 15 Oct 2024
It's Just Another Day: Unique Video Captioning by Discriminative Prompting Toby Perrett Tengda Han Dima Damen Andrew Zisserman 24 3 0 15 Oct 2024
VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models Xiaohan Lan Yitian Yuan Zequn Jie Lin Ma VLM 57 2 0 15 Oct 2024
Character-aware audio-visual subtitling in context Jaesung Huh Andrew Zisserman 41 0 0 14 Oct 2024
When Does Perceptual Alignment Benefit Vision Representations? Shobhita Sundaram Stephanie Fu Lukas Muttenthaler Netanel Y. Tamir Lucy Chai Simon Kornblith Trevor Darrell Phillip Isola 57 7 1 14 Oct 2024
Free Video-LLM: Prompt-guided Visual Perception for Efficient Training-free Video LLMs Kai Han Jianyuan Guo Yehui Tang W. He Enhua Wu Yunhe Wang MLLM VLM 21 3 0 14 Oct 2024
Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models Juseong Jin Chang Wook Jeong 33 3 0 13 Oct 2024
Towards Efficient Visual-Language Alignment of the Q-Former for Visual Reasoning Tasks Sungkyung Kim Adam Lee Junyoung Park Andrew Chung Jusang Oh Jay-Yoon Lee 26 3 0 12 Oct 2024
Audio Description Generation in the Era of LLMs and VLMs: A Review of Transferable Generative AI Technologies Yingqiang Gao Lukas Fischer Alexa Lintner Sarah Ebling 36 0 0 11 Oct 2024
Sample then Identify: A General Framework for Risk Control and Assessment in Multimodal Large Language Models Qingni Wang Tiantian Geng Zhiyuan Wang Teng Wang Bo Fu Feng Zheng 40 5 0 10 Oct 2024
$G$^{2}$TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models$ G $^{2}$ TR: Generalized Grounded Temporal Reasoning for Robot Instruction Following by Combining Large Pre-trained Models Riya Arora N. N. Aman Tambi Sandeep S. Zachariah Souvik Chakraborty Rohan Paul LM&Ro 31 0 0 10 Oct 2024
Enhancing Multimodal LLM for Detailed and Accurate Video Captioning using Multi-Round Preference Optimization Changli Tang Yixuan Li Yudong Yang Jimin Zhuang Guangzhi Sun Wei Li Zejun Ma Chao Zhang 33 4 0 09 Oct 2024
MM-Ego: Towards Building Egocentric Multimodal LLMs for Video QA Hanrong Ye Haotian Zhang Erik Daxberger Lin Chen Zongyu Lin ... Haoxuan You Dan Xu Zhe Gan Jiasen Lu Yinfei Yang EgoV MLLM 88 12 0 09 Oct 2024
Temporal Reasoning Transfer from Text to Video Lei Li Yuanxin Liu Linli Yao Peiyuan Zhang Chenxin An Lean Wang Xu Sun Lingpeng Kong Qi Liu LRM 48 7 0 08 Oct 2024
Grounding is All You Need? Dual Temporal Grounding for Video Dialog You Qin Wei Ji Xinze Lan Hao Fei Xun Yang Dan Guo Roger Zimmermann Lizi Liao VGen 48 0 0 08 Oct 2024
Enhancing Temporal Modeling of Video LLMs via Time Gating Zi-Yuan Hu Yiwu Zhong Shijia Huang M. Lyu Liwei Wang VLM 33 0 0 08 Oct 2024
Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond Soyeon Caren Han Feiqi Cao Josiah Poon Roberto Navigli MLLM VLM 45 5 0 08 Oct 2024
TRACE: Temporal Grounding Video LLM via Causal Event Modeling Yongxin Guo Jingyu Liu Mingda Li Xiaoying Tang Qingbin Liu Xiaoying Tang 52 14 0 08 Oct 2024
Polymath: A Challenging Multi-modal Mathematical Reasoning Benchmark Himanshu Gupta Shreyas Verma Ujjwala Anantheswaran Kevin Scaria Mihir Parmar Swaroop Mishra Chitta Baral ReLM LRM 32 7 0 06 Oct 2024
Realizing Video Summarization from the Path of Language-based Semantic Understanding Kuan-Chen Mu Zhi-Yi Chin Wei-Chen Chiu 28 0 0 06 Oct 2024
Grounded-VideoLLM: Sharpening Fine-grained Temporal Grounding in Video Large Language Models Haibo Wang Zhiyang Xu Yu Cheng Shizhe Diao Yufan Zhou Yixin Cao Qifan Wang Weifeng Ge Lifu Huang 24 21 0 04 Oct 2024
StoryNavi: On-Demand Narrative-Driven Reconstruction of Video Play With Generative AI Alston Lantian Xu Tianwei Ma Tianmeng Liu Can Liu Alvaro Cassinelli VGen 39 0 0 04 Oct 2024
Self-Powered LLM Modality Expansion for Large Speech-Text Models Tengfei Yu Xuebo Liu Zhiyi Hou Liang Ding Dacheng Tao Min Zhang 32 0 0 04 Oct 2024
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 89 26 0 04 Oct 2024
Frame-Voyager: Learning to Query Frames for Video Large Language Models Sicheng Yu Chengkai Jin Huanyu Wang Zhenghao Chen Sheng Jin ... Zhenbang Sun Bingni Zhang Jiawei Wu Hao Zhang Qianru Sun 77 5 0 04 Oct 2024
Video Instruction Tuning With Synthetic Data Yuanhan Zhang Jinming Wu Wei Li Bo Li Zejun Ma Ziwei Liu Chunyuan Li SyDa VGen 55 144 0 03 Oct 2024
Visual Prompting in LLMs for Enhancing Emotion Recognition Qixuan Zhang Zhifeng Wang Dylan Zhang Wenjia Niu Sabrina Caldwell Tom Gedeon Yang Liu Zhenyue Qin 37 0 0 03 Oct 2024
Open-vocabulary Multimodal Emotion Recognition: Dataset, Metric, and Benchmark Zheng Lian Haiyang Sun Guoying Zhao Lan Chen Haoyu Chen ... Rui Liu Shan Liang Ya Li Jiangyan Yi Jianhua Tao VLM 43 0 0 02 Oct 2024
UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark Hasnat Md Abdullah Tian Liu Kangda Wei Shu Kong Ruihong Huang 44 3 0 02 Oct 2024
VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models Jiapeng Wang Chengyu Wang Kunzhe Huang Jun Huang Lianwen Jin CLIP VLM 45 3 0 01 Oct 2024
Efficient Driving Behavior Narration and Reasoning on Edge Device Using Large Language Models Yizhou Huang Yihua Cheng Kezhi Wang LRM 52 1 0 30 Sep 2024
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos Zechen Bai Tong He Haiyang Mei Pichao Wang Ziteng Gao Joya Chen Lei Liu Zheng Zhang Mike Zheng Shou VLM VOS MLLM 50 17 0 29 Sep 2024
Video DataFlywheel: Resolving the Impossible Data Trinity in Video-Language Understanding Xiao Wang Jianlong Wu Zijia Lin Fuzheng Zhang Di Zhang Liqiang Nie VGen 37 1 0 29 Sep 2024
From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding Heqing Zou Tianze Luo Guiyang Xie Victor Zhang ... Guangcong Wang Juanyang Chen Zhuochen Wang Hansheng Zhang Huaijian Zhang VLM 45 6 0 27 Sep 2024
EgoLM: Multi-Modal Language Model of Egocentric Motions Fangzhou Hong Vladimir Guzov Hyo Jin Kim Yuting Ye Richard Newcombe Ziwei Liu Lingni Ma 45 4 0 26 Sep 2024
E.T. Bench: Towards Open-Ended Event-Level Video-Language Understanding Ye Liu Zongyang Ma Zhongang Qi Yang Wu Ying Shan Chang Wen Chen 41 16 0 26 Sep 2024
LLM4Brain: Training a Large Language Model for Brain Video Understanding Ruizhe Zheng Lichao Sun 29 0 0 26 Sep 2024
MECD: Unlocking Multi-Event Causal Discovery in Video Reasoning Tieyuan Chen Huabin Liu Tianyao He Yihang Chen Chaofan Gan ... Cheng Zhong Yang Zhang Yingxue Wang Hui Lin Weiyao Lin VGen CML 41 5 0 26 Sep 2024
EAGLE: Egocentric AGgregated Language-video Engine Jing Bi Yunlong Tang Luchuan Song A. Vosoughi Nguyen Nguyen Chenliang Xu 53 8 0 26 Sep 2024