OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

29 November 2023

Conghui He

Dahua Lin

Papers citing "OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation"

50 / 142 papers shown

Title
VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding Jiaqi Wang Yifei Gao Jitao Sang MLLM 123 2 0 24 Nov 2024
Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens Zhangqi Jiang Junkai Chen Beier Zhu Tingjin Luo Yankun Shen Xu Yang 106 4 0 23 Nov 2024
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models Junzhe Chen Tianshu Zhang S. Huang Yuwei Niu Linfeng Zhang Lijie Wen Xuming Hu MLLM VLM 204 2 0 22 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie Xingchen Sun Zhanhui Kang Xirong Li MLLM 35 4 0 15 Nov 2024
DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark Haodong Li Haicheng Qu Xiaofeng Zhang 43 1 0 05 Nov 2024
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 60 13 0 05 Nov 2024
Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models J. Wu Tsz Ting Chung Kai Chen Dit-Yan Yeung VLM LRM 65 3 0 30 Oct 2024
Are LLM-Judges Robust to Expressions of Uncertainty? Investigating the effect of Epistemic Markers on LLM-based Evaluation Dongryeol Lee Yerin Hwang Yongil Kim Joonsuk Park Kyomin Jung ELM 72 5 0 28 Oct 2024
PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reduction Long Xing Qidong Huang Xiaoyi Dong Jiajie Lu Pan Zhang ... Yuhang Cao Zeang Sheng Jiaqi Wang Feng Wu Dahua Lin VLM 50 28 0 22 Oct 2024
LLMScan: Causal Scan for LLM Misbehavior Detection Mengdi Zhang Kai Kiat Goh Peixin Zhang Jun Sun Rose Lin Xin Hongyu Zhang 25 0 0 22 Oct 2024
Mitigating Object Hallucination via Concentric Causal Attention Yun Xing Yiheng Li Ivan Laptev Shijian Lu 53 18 0 21 Oct 2024
Reducing Hallucinations in Vision-Language Models via Latent Space Steering Sheng Liu Haotian Ye Lei Xing James Zou VLM LLMSV 50 5 0 21 Oct 2024
A Survey of Hallucination in Large Visual Language Models Wei Lan Wenyi Chen Qingfeng Chen Shirui Pan Huiyu Zhou Yi-Lun Pan LRM 33 4 0 20 Oct 2024
Modality-Fair Preference Optimization for Trustworthy MLLM Alignment Songtao Jiang Yan Zhang Ruizhe Chen Yeying Jin Zuozhu Liu MLLM MoE 28 6 0 20 Oct 2024
Trust but Verify: Programmatic VLM Evaluation in the Wild Viraj Prabhu Senthil Purushwalkam An Yan Caiming Xiong Ran Xu MLLM 38 1 0 17 Oct 2024
The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio Sicong Leng Yun Xing Zesen Cheng Yang Zhou Hang Zhang Xin Li Deli Zhao Shijian Lu Chunyan Miao Lidong Bing 40 8 0 16 Oct 2024
A Theoretical Survey on Foundation Models Shi Fu Yuzhu Chen Yingjie Wang Dacheng Tao 28 0 0 15 Oct 2024
MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation Chenxi Wang Xiang Chen N. Zhang Bozhong Tian Haoming Xu Shumin Deng H. Chen MLLM LRM 37 4 0 15 Oct 2024
LongHalQA: Long-Context Hallucination Evaluation for MultiModal Large Language Models Han Qiu Jiaxing Huang Peng Gao Qin Qi Xiaoqin Zhang Ling Shao Shijian Lu HILM 33 1 0 13 Oct 2024
Insight Over Sight? Exploring the Vision-Knowledge Conflicts in Multimodal LLMs Xiaoyuan Liu Wenxuan Wang Youliang Yuan Jen-tse Huang Qiuzhi Liu Pinjia He Zhaopeng Tu 155 1 0 10 Oct 2024
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models Yuying Shang Xinyi Zeng Yutao Zhu Xiao Yang Zhengwei Fang Jingyuan Zhang Jiawei Chen Zinan Liu Yu Tian VLM MLLM 138 1 0 09 Oct 2024
Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causality Guanyu Zhou Yibo Yan Xin Zou Kun Wang Aiwei Liu Xuming Hu 51 7 0 07 Oct 2024
Differential Transformer Tianzhu Ye Li Dong Yuqing Xia Yutao Sun Yi Zhu Gao Huang Furu Wei 171 0 0 07 Oct 2024
DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination Xuan Gong Tianshi Ming Xinpeng Wang Zhihua Wei MLLM 42 10 0 06 Oct 2024
SELU: Self-Learning Embodied MLLMs in Unknown Environments Boyu Li Haobin Jiang Ziluo Ding Xinrun Xu Haoran Li Dongbin Zhao Zongqing Lu LRM 52 2 0 04 Oct 2024
Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models Xin Zou Yizhou Wang Yibo Yan Yuanhuiyi Lyu Kening Zheng ... Junkai Chen Peijie Jiang Jiaheng Liu Chang Tang Xuming Hu 89 7 0 04 Oct 2024
Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations Nick Jiang Anish Kachinthaya Suzie Petryk Yossi Gandelsman VLM 34 16 0 03 Oct 2024
HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding Fan Yuan Chi Qin Xiaogang Xu Piji Li VLM MLLM 33 4 0 30 Sep 2024
A Unified Hallucination Mitigation Framework for Large Vision-Language Models Yue Chang Liqiang Jing Xiaopeng Zhang Yue Zhang VLM MLLM 68 2 0 24 Sep 2024
Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding Xiaoyu Liang Jiayuan Yu Lianrui Mu Jiedong Zhuang Jiaqi Hu Yuchen Yang Jiangnan Ye Lu Lu Jian Chen Haoji Hu VLM 45 2 0 10 Sep 2024
Understanding Multimodal Hallucination with Parameter-Free Representation Alignment Yueqian Wang Jianxin Liang Yuxuan Wang Huishuai Zhang Dongyan Zhao 46 1 0 02 Sep 2024
Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data Spencer Whitehead Jacob Phillips Sean Hendryx 31 0 0 30 Aug 2024
Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning Xiaoye Qu Jiashuo Sun Wei Wei Yu Cheng MLLM LRM 34 14 0 30 Aug 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 32 8 0 28 Aug 2024
ConVis: Contrastive Decoding with Hallucination Visualization for Mitigating Hallucinations in Multimodal Large Language Models Yeji Park Deokyeong Lee Junsuk Choe Buru Chang MLLM VLM 64 5 0 25 Aug 2024
RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data Chenglong Wang Yang Gan Yifu Huo Yongyu Mu Murun Yang ... Chunliang Zhang Tongran Liu Quan Du Di Yang Jingbo Zhu VLM 71 4 0 22 Aug 2024
Towards Analyzing and Mitigating Sycophancy in Large Vision-Language Models Yunpu Zhao Rui Zhang Junbin Xiao Changxin Ke Ruibo Hou Yifan Hao Qi Guo Yunji Chen 31 4 0 21 Aug 2024
Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models Kening Zheng Junkai Chen Yibo Yan Xin Zou Xuming Hu 113 6 0 18 Aug 2024
Reference-free Hallucination Detection for Large Vision-Language Models Qing Li Chenyang Lyu Jiahui Geng Derui Zhu Maxim Panov Fakhri Karray 26 6 0 11 Aug 2024
Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models Fushuo Huo Wenchao Xu Zhong Zhang Yining Qi Zhicheng Chen Peilin Zhao VLM MLLM 66 19 0 04 Aug 2024
Hallu-PI: Evaluating Hallucination in Multi-modal Large Language Models within Perturbed Inputs Peng Ding Jingyu Wu Mark Girolami Dan Ma Xuezhi Cao Xunliang Cai Shi Chen T. J. Sullivan Shujian Huang AAML VLM MLLM 39 51 0 02 Aug 2024
Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation Cephas Mpungu Qiyuan Chen Wei Wei Jiashuo Sun G. Mapp VLM RALM LRM 33 16 0 01 Aug 2024
Mitigating Multilingual Hallucination in Large Vision-Language Models Xiaoye Qu Mingyang Song Wei Wei Jianfeng Dong Yu Cheng VLM LRM 37 2 0 01 Aug 2024
Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs Shiping Liu Kecheng Zheng Wei Chen MLLM 49 34 0 31 Jul 2024
DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer Jinfeng Wei Xiaofeng Zhang 28 13 0 21 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 60 5 0 11 Jul 2024
Multi-Object Hallucination in Vision-Language Models Xuweiyi Chen Ziqiao Ma Xuejun Zhang Sihan Xu Shengyi Qian Jianing Yang David Fouhey Joyce Chai 49 16 0 08 Jul 2024
RULE: Reliable Multimodal RAG for Factuality in Medical Vision Language Models Peng Xia Kangyu Zhu Haoran Li Hongtu Zhu Yun-Qing Li Gang Li Linjun Zhang Huaxiu Yao MedIm 42 32 0 06 Jul 2024
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? Zhaorun Chen Yichao Du Zichen Wen Yiyang Zhou Chenhang Cui ... Jiawei Zhou Zhuokai Zhao Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM MLLM 63 29 0 05 Jul 2024
MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs Wenqian Ye Guangtao Zheng Yunsheng Ma Xu Cao Bolin Lai James M. Rehg Aidong Zhang 37 10 0 24 Jun 2024