Hallucination of Multimodal Large Language Models: A Survey

29 April 2024

Tianjun Xiao

Zheng Zhang

Papers citing "Hallucination of Multimodal Large Language Models: A Survey"

50 / 115 papers shown

Title
MovieBench: A Hierarchical Movie Level Dataset for Long Video Generation Weijia Wu Mingyu Liu Zeyu Zhu Xi Xia Haoen Feng Wen Wang Kevin Qinghong Lin Chunhua Shen Mike Zheng Shou DiffM VGen 116 1 0 22 Nov 2024
Understanding Multimodal LLMs: the Mechanistic Interpretability of Llava in Visual Question Answering Zeping Yu Sophia Ananiadou 124 0 0 17 Nov 2024
Thinking Before Looking: Improving Multimodal LLM Reasoning via Mitigating Visual Hallucination Haojie Zheng Tianyang Xu Hanchi Sun Shu Pu Ruoxi Chen Lichao Sun MLLM LRM 79 8 0 15 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie X. Sun Zhanhui Kang Xirong Li MLLM 33 3 0 15 Nov 2024
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 57 13 0 05 Nov 2024
Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios Yunkai Dang Mengxi Gao Yibo Yan Xin Zou Yanggan Gu Aiwei Liu Xuming Hu 42 4 0 05 Nov 2024
Benchmarking Multimodal Retrieval Augmented Generation with Dynamic VQA Dataset and Self-adaptive Planning Agent Yangning Li Yinghui Li Xinyu Wang Yong-feng Jiang Zhen Zhang ... Hui Wang Hai-Tao Zheng Pengjun Xie Philip S. Yu Fei Huang 62 15 0 05 Nov 2024
RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models Serena Zhang S. Oishi Banerjee J. N. Acosta L. John Fahrner Pranav Rajpurkar VLM 28 0 0 01 Nov 2024
Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map Xinyuan Chang Maixuan Xue Xinran Liu Zheng Pan Xing Wei 48 1 0 31 Oct 2024
LLM-Consensus: Multi-Agent Debate for Visual Misinformation Detection Kumud Lakara Juil Sock Christian Rupprecht Philip H. S. Torr John Collomosse Christian Schroeder de Witt Christian Schroeder de Witt 27 2 0 26 Oct 2024
A Survey of Hallucination in Large Visual Language Models Wei Lan Wenyi Chen Qingfeng Chen Shirui Pan Huiyu Zhou Yi-Lun Pan LRM 30 4 0 20 Oct 2024
Modality-Fair Preference Optimization for Trustworthy MLLM Alignment Songtao Jiang Yan Zhang Ruizhe Chen Yeying Jin Zuozhu Liu MLLM MoE 26 6 0 20 Oct 2024
MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation Chenxi Wang Xiang Chen N. Zhang Bozhong Tian Haoming Xu Shumin Deng H. Chen MLLM LRM 29 4 0 15 Oct 2024
LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models Han Qiu Jiaxing Huang Peng Gao Qin Qi Xiaoqin Zhang Ling Shao Shijian Lu HILM 25 1 0 13 Oct 2024
ProcBench: Benchmark for Multi-Step Reasoning and Following Procedure Ippei Fujisawa Sensho Nobe Hiroki Seto Rina Onda Yoshiaki Uchida Hiroki Ikoma Pei-Chun Chien Ryota Kanai LRM 39 3 0 04 Oct 2024
Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models Xin Zou Yizhou Wang Yibo Yan Yuanhuiyi Lyu Kening Zheng ... Junkai Chen Peijie Jiang J. Liu Chang Tang Xuming Hu 83 7 0 04 Oct 2024
From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities Wanpeng Zhang Zilong Xie Yicheng Feng Yijiang Li Xingrun Xing Sipeng Zheng Zongqing Lu MLLM 30 0 0 03 Oct 2024
Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering K. K. Bernhard Schölkopf Michael Muehlebach 26 0 0 02 Oct 2024
Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliability Weitong Zhang Chengqi Zang Bernhard Kainz 28 0 0 01 Oct 2024
One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos Zechen Bai Tong He Haiyang Mei Pichao Wang Ziteng Gao Joya Chen Lei Liu Zheng Zhang Mike Zheng Shou VLM VOS MLLM 45 17 0 29 Sep 2024
A Survey on the Honesty of Large Language Models Siheng Li Cheng Yang Taiqiang Wu Chufan Shi Yuji Zhang ... Jie Zhou Yujiu Yang Ngai Wong Xixin Wu Wai Lam HILM 29 4 0 27 Sep 2024
A Unified Hallucination Mitigation Framework for Large Vision-Language Models Yue Chang Liqiang Jing Xiaopeng Zhang Yue Zhang VLM MLLM 58 2 0 24 Sep 2024
A Survey on Multimodal Benchmarks: In the Era of Large AI Models Lin Li Guikun Chen Hanrong Shi Jun Xiao Long Chen 39 9 0 21 Sep 2024
Towards Child-Inclusive Clinical Video Understanding for Autism Spectrum Disorder Aditya Kommineni Digbalay Bose Tiantian Feng So Hyun Kim Helen Tager-Flusberg Somer Bishop C. Lord Sudarsana Reddy Kadiri Shrikanth Narayanan 35 1 0 20 Sep 2024
Surveying the MLLM Landscape: A Meta-Review of Current Surveys Ming Li Keyu Chen Ziqian Bi Ming Liu Benji Peng ... Jinlang Wang Sen Zhang X. Pan Jiawei Xu Pohsun Feng OffRL 39 2 0 17 Sep 2024
Measuring and Enhancing Trustworthiness of LLMs in RAG through Grounded Attributions and Learning to Refuse Maojia Song Shang Hong Sim Rishabh Bhardwaj Hai Leong Chieu Navonil Majumder Soujanya Poria 29 6 0 17 Sep 2024
When Context Leads but Parametric Memory Follows in Large Language Models Yufei Tao Adam Hiatt Erik Haake Antonie J. Jetter Ameeta Agrawal KELM 38 0 0 13 Sep 2024
Visual Prompting in Multimodal Large Language Models: A Survey Junda Wu Zhehao Zhang Yu Xia Xintong Li Zhaoyang Xia ... Subrata Mitra Dimitris N. Metaxas Lina Yao Jingbo Shang Julian McAuley VLM LRM 50 12 0 05 Sep 2024
Understanding Multimodal Hallucination with Parameter-Free Representation Alignment Yueqian Wang Jianxin Liang Yuxuan Wang Huishuai Zhang Dongyan Zhao 41 1 0 02 Sep 2024
Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey Qika Lin Yifan Zhu Xin Mei Ling Huang Jingying Ma Kai He Zhen Peng Erik Cambria Mengling Feng 34 17 0 23 Aug 2024
Show-o: One Single Transformer to Unify Multimodal Understanding and Generation Jinheng Xie Weijia Mao Zechen Bai David Junhao Zhang Weihao Wang Kevin Qinghong Lin Yuchao Gu Zhijie Chen Zhenheng Yang Mike Zheng Shou 46 160 0 22 Aug 2024
CoVLA: Comprehensive Vision-Language-Action Dataset for Autonomous Driving Hidehisa Arai Keita Miwa Kento Sasaki Yu Yamaguchi Kohei Watanabe Shunsuke Aoki Issei Yamamoto 40 9 0 19 Aug 2024
Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models Kening Zheng Junkai Chen Yibo Yan Xin Zou Xuming Hu 99 6 0 18 Aug 2024
VideoQA in the Era of LLMs: An Empirical Study Junbin Xiao Nanxin Huang Hangyu Qin Dongyang Li Yicong Li ... Zhulin Tao Jianxing Yu Liang Lin Tat-Seng Chua Angela Yao 23 10 0 08 Aug 2024
Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey Atsuyuki Miyai Jingkang Yang Jingyang Zhang Yifei Ming Sisir Dhakal ... Yixuan Li Hai Li Ziwei Liu Toshihiko Yamasaki Kiyoharu Aizawa 38 9 0 31 Jul 2024
Interpreting and Mitigating Hallucination in MLLMs through Multi-agent Debate Zheng Lin Zhenxing Niu Zhibin Wang Yinghui Xu 31 4 0 30 Jul 2024
CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models Junda Wu Xintong Li Tong Yu Yu-Xiang Wang Xiang Chen Jiuxiang Gu Lina Yao Jingbo Shang Julian McAuley 39 0 0 29 Jul 2024
INF-LLaVA: Dual-perspective Perception for High-Resolution Multimodal Large Language Model Yiwei Ma Zhibin Wang Xiaoshuai Sun Weihuang Lin Qiang-feng Zhou Jiayi Ji Rongrong Ji MLLM VLM 49 1 0 23 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 48 5 0 11 Jul 2024
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models Jinliang Lu Ziliang Pang Min Xiao Yaochen Zhu Rui Xia Jiajun Zhang MoMe 38 18 0 08 Jul 2024
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification Pritish Sahu Karan Sikka Ajay Divakaran MLLM LRM 64 4 0 02 Jul 2024
Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness Khyathi Raghavi Chandu Linjie Li Anas Awadalla Ximing Lu Jae Sung Park Jack Hessel Lijuan Wang Yejin Choi 45 2 0 02 Jul 2024
MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval Weitong Cai Jiabo Huang Shaogang Gong Hailin Jin Yang Liu 39 0 0 25 Jun 2024
Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models Bei Yan Jie Zhang Zheng Yuan Shiguang Shan Xilin Chen VLM 35 4 0 24 Jun 2024
SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model Yongting Zhang Lu Chen Guodong Zheng Yifeng Gao Rui Zheng ... Yu Qiao Xuanjing Huang Feng Zhao Tao Gui Jing Shao VLM 79 23 0 17 Jun 2024
Detecting and Evaluating Medical Hallucinations in Large Vision Language Models Jiawei Chen Dingkang Yang Tong Wu Yue Jiang Xiaolu Hou Mingcheng Li Shunli Wang Dongling Xiao Ke Li Lihua Zhang LM&MA VLM 31 18 0 14 Jun 2024
Vision-Language Models Meet Meteorology: Developing Models for Extreme Weather Events Detection with Heatmaps Jian Chen Peilin Zhou Yining Hua Dading Chong Meng Cao Yaowei Li Zixuan Yuan Bing Zhu Junwei Liang VLM 41 1 0 14 Jun 2024
Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM) Anjanava Biswas Wrick Talukdar 23 7 0 13 Jun 2024
CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks Maciej Besta Lorenzo Paleari Aleš Kubíček Piotr Nyczyk Robert Gerstenberger Patrick Iff Tomasz Lehmann H. Niewiadomski Torsten Hoefler 58 5 0 04 Jun 2024
CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models Junho Kim Hyunjun Kim Yeonju Kim Yong Man Ro MLLM 39 10 0 04 Jun 2024