v1v2 (latest)

Hallucination of Multimodal Large Language Models: A Survey

29 April 2024

Tianjun Xiao

Zheng Zhang

Papers citing "Hallucination of Multimodal Large Language Models: A Survey"

50 / 261 papers shown

Title
Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Models Jinliang Lu Ziliang Pang Min Xiao Yaochen Zhu Rui Xia Jiajun Zhang MoMe 119 27 0 08 Jul 2024
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification Pritish Sahu Karan Sikka Ajay Divakaran MLLM LRM 109 6 0 02 Jul 2024
Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness Khyathi Chandu Linjie Li Anas Awadalla Ximing Lu Jae Sung Park Jack Hessel Lijuan Wang Yejin Choi 103 3 0 02 Jul 2024
Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models Weihong Zhong Xiaocheng Feng Liang Zhao Qiming Li Lei Huang Yuxuan Gu Weitao Ma Yuan Xu Bing Qin MLLM 115 11 0 30 Jun 2024
MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval Weitong Cai Jiabo Huang Shaogang Gong Hailin Jin Yang Liu 83 0 0 25 Jun 2024
Evaluating the Quality of Hallucination Benchmarks for Large Vision-Language Models Bei Yan Jie Zhang Zheng Yuan Shiguang Shan Xilin Chen VLM 68 5 0 24 Jun 2024
SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model Yongting Zhang Lu Chen Guodong Zheng Yifeng Gao Rui Zheng ... Yu Qiao Xuanjing Huang Feng Zhao Tao Gui Jing Shao VLM 228 33 0 17 Jun 2024
Detecting and Evaluating Medical Hallucinations in Large Vision Language Models Jiawei Chen Dingkang Yang Tong Wu Yue Jiang Xiaolu Hou Mingcheng Li Shunli Wang Dongling Xiao Ke Li Lihua Zhang LM&MA VLM 80 22 0 14 Jun 2024
ClimateIQA: A New Dataset and Benchmark to Advance Vision-Language Models in Meteorology Anomalies Analysis Jian Chen Peilin Zhou Yining Hua Dading Chong Meng Cao Yaowei Li Wei Chen Bing Zhu Junwei Liang Zixuan Yuan VLM 110 3 0 14 Jun 2024
Robustness of Structured Data Extraction from In-plane Rotated Documents using Multi-Modal Large Language Models (LLM) Anjanava Biswas Wrick Talukdar 65 8 0 13 Jun 2024
CODE: Contrasting Self-generated Description to Combat Hallucination in Large Multi-modal Models Junho Kim Hyunjun Kim Yeonju Kim Yong Man Ro MLLM 117 16 0 04 Jun 2024
CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks Maciej Besta Lorenzo Paleari Marcin Copik Robert Gerstenberger Aleš Kubíček ... Eric Schreiber Torsten Hoefler Tomasz Lehmann H. Niewiadomski Torsten Hoefler 171 7 0 04 Jun 2024
RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in LVLMs Sangmin Woo Jaehyuk Jang Donguk Kim Yubin Choi Changick Kim 67 1 0 28 May 2024
Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song Hengtao Shen MLLM 187 12 0 24 May 2024
Visual Fact Checker: Enabling High-Fidelity Detailed Caption Generation Yunhao Ge Fangyin Wei Siddharth Gururani Nayeon Lee Xuan Li Huayu Chen CoGe DiffM 72 17 0 30 Apr 2024
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs An Yan Zhengyuan Yang Junda Wu Wanrong Zhu Jianwei Yang ... Kevin Qinghong Lin Jianfeng Wang Julian McAuley Jianfeng Gao Lijuan Wang LRM 112 12 0 25 Apr 2024
Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback Wenyi Xiao Ziwei Huang Leilei Gan Wanggui He Haoyuan Li Zhelun Yu Hao Jiang Leilei Gan Linchao Zhu MLLM 104 34 0 22 Apr 2024
FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedback Liqiang Jing Xinya Du 187 18 0 07 Apr 2024
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding Xintong Wang Jingheng Pan Liang Ding Christian Biemann MLLM 126 75 0 27 Mar 2024
Multi-Modal Hallucination Control by Visual Information Grounding Alessandro Favero Luca Zancato Matthew Trager Siddharth Choudhary Pramuditha Perera Alessandro Achille Ashwin Swaminathan Stefano Soatto MLLM 132 80 0 20 Mar 2024
What if...?: Thinking Counterfactual Keywords Helps to Mitigate Hallucination in Large Multi-modal Models Junho Kim Yeonju Kim Yonghyun Ro LRM MLLM 68 5 0 20 Mar 2024
Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning Dongmin Park Zhaofang Qian Guangxing Han Ser-Nam Lim MLLM 84 0 0 15 Mar 2024
HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding Zhaorun Chen Zhuokai Zhao Hongyin Luo Huaxiu Yao Bo Li Jiawei Zhou MLLM 121 75 0 01 Mar 2024
IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding Lanyun Zhu Deyi Ji Tianrun Chen Peng Xu Jieping Ye Jun Liu MLLM 115 53 0 28 Feb 2024
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning Yiyang Zhou Chenhang Cui Rafael Rafailov Chelsea Finn Huaxiu Yao VLM MLLM 120 121 0 18 Feb 2024
EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models Shangyu Xing Fei Zhao Zhen Wu Tuo An Weihao Chen Chunhui Li Jianbing Zhang Xinyu Dai MLLM MU 123 5 0 15 Feb 2024
Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance Linxi Zhao Yihe Deng Weitong Zhang Q. Gu MLLM 105 1 0 13 Feb 2024
ViGoR: Improving Visual Grounding of Large Vision Language Models with Fine-Grained Reward Modeling Siming Yan Min Bai Weifeng Chen Xiong Zhou Qixing Huang Erran L. Li VLM 50 20 0 09 Feb 2024
The Instinctive Bias: Spurious Images lead to Hallucination in MLLMs Tianyang Han Qing Lian Boyao Wang Renjie Pi Jipeng Zhang Shizhe Diao Yong Lin Tong Zhang 98 1 0 06 Feb 2024
Enhancing Multimodal Large Language Models with Vision Detection Models: An Empirical Study Qirui Jiao Daoyuan Chen Yilun Huang Yaliang Li Ying Shen 56 15 0 31 Jan 2024
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs Shengbang Tong Zhuang Liu Yuexiang Zhai Yi-An Ma Yann LeCun Saining Xie VLM MLLM 160 349 0 11 Jan 2024
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 288 1,216 0 21 Dec 2023
VCoder: Versatile Vision Encoders for Multimodal Large Language Models Jitesh Jain Jianwei Yang Humphrey Shi MLLM 76 31 0 21 Dec 2023
Silkie: Preference Distillation for Large Visual Language Models Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang Liang Chen Yazheng Yang Benyou Wang Lingpeng Kong MLLM 183 80 0 17 Dec 2023
Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites Lei Wang Jiabang He Shenshen Li Ning Liu Ee-Peng Lim MLLM 90 43 0 04 Dec 2023
Behind the Magic, MERLIM: Multi-modal Evaluation Benchmark for Large Image-Language Models Andrés Villa Juan Carlos León Alcázar Alvaro Soto Bernard Ghanem MLLM VLM 85 11 0 03 Dec 2023
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback M. Steyvers Yuan Yao Haoye Zhang Taiwen He Yifeng Han ... Xinyue Hu Zhiyuan Liu Hai-Tao Zheng Maosong Sun Tat-Seng Chua MLLM VLM 225 230 0 01 Dec 2023
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation Qidong Huang Xiao-wen Dong Pan Zhang Bin Wang Conghui He Jiaqi Wang Dahua Lin Weiming Zhang Neng H. Yu MLLM 148 206 0 29 Nov 2023
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding Sicong Leng Hang Zhang Guanzheng Chen Xin Li Shijian Lu Chunyan Miao Li Bing VLM MLLM 153 239 0 28 Nov 2023
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization Zhiyuan Zhao Bin Wang Linke Ouyang Xiao-wen Dong Jiaqi Wang Conghui He MLLM VLM 143 135 0 28 Nov 2023
Mitigating Hallucination in Visual Language Models with Visual Supervision Zhiyang Chen Yousong Zhu Yufei Zhan Zhaowen Li Chaoyang Zhao Jinqiao Wang Ming Tang VLM MLLM 114 33 0 27 Nov 2023
Multimodal Large Language Models: A Survey Jiayang Wu Wensheng Gan Zefeng Chen Shicheng Wan Philip S. Yu 95 196 0 22 Nov 2023
HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data Qifan Yu Juncheng Li Longhui Wei Liang Pang Wentao Ye Bosheng Qin Siliang Tang Qi Tian Yueting Zhuang MLLM VLM 121 82 0 22 Nov 2023
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models Ziyi Lin Chris Liu Renrui Zhang Peng Gao Longtian Qiu ... Siyuan Huang Yichi Zhang Xuming He Hongsheng Li Yu Qiao MLLM VLM 115 231 0 13 Nov 2023
AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation Junyang Wang Yuhang Wang Guohai Xu Jing Zhang Yukai Gu ... Jiaqi Wang Haiyang Xu Ming Yan Ji Zhang Jitao Sang MLLM VLM 98 122 0 13 Nov 2023
InfMLLM: A Unified Framework for Visual-Language Tasks Qiang-feng Zhou Zhibin Wang Wei Chu Yinghui Xu Hao Li Yuan Qi MLLM 60 12 0 12 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 260 422 0 07 Nov 2023
Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challenges Chenhang Cui Yiyang Zhou Xinyu Yang Shirley Wu Linjun Zhang James Zou Huaxiu Yao MLLM 98 93 0 06 Nov 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 243 2,832 0 05 Oct 2023
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models Yiyang Zhou Chenhang Cui Jaehong Yoon Linjun Zhang Zhun Deng Chelsea Finn Mohit Bansal Huaxiu Yao MLLM 167 186 0 01 Oct 2023