Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization

28 November 2023

Conghui He

Papers citing "Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization"

50 / 94 papers shown

Title
Critique Before Thinking: Mitigating Hallucination through Rationale-Augmented Instruction Tuning Zexian Yang Dian Li Dayan Wu Gang Liu Weiping Wang MLLM LRM 41 0 0 12 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 76 2 0 05 May 2025
Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models Sangmin Woo Kang Zhou Yun Zhou Shuai Wang Sheng Guan Haibo Ding Lin Lee Cheong VPVLM 84 0 0 30 Apr 2025
Antidote: A Unified Framework for Mitigating LVLM Hallucinations in Counterfactual Presupposition and Object Perception Yuanchen Wu Lu Zhang Hang Yao Junlong Du Ke Yan Shouhong Ding Yunsheng Wu Xuzhao Li MLLM 73 0 0 29 Apr 2025
AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization Jinda Lu Jinghan Li Yuan Gao Junkang Wu Jiancan Wu Xuben Wang Xiangnan He 195 0 0 22 Apr 2025
VistaDPO: Video Hierarchical Spatial-Temporal Direct Preference Optimization for Large Video Models Haojian Huang Haodong Chen Shengqiong Wu Meng Luo Jinlan Fu Xinya Du Han Zhang Hao Fei AI4TS 229 1 0 17 Apr 2025
TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection C. Xie Tongxuan Liu Lei Jiang Yuting Zeng J. Guo Yunheng Shen Weizhe Huang Jing Li Xiaohua Xu VLM 61 0 0 05 Apr 2025
Q-Insight: Understanding Image Quality via Visual Reinforcement Learning Weiqi Li X. Zhang Shijie Zhao Yuyao Zhang Junlin Li Li Zhang Jian Zhang 50 4 0 28 Mar 2025
Instruction-Oriented Preference Alignment for Enhancing Multi-Modal Comprehension Capability of MLLMs Zitian Wang Yue Liao Kang Rong Fengyun Rao Yibo Yang Si Liu 77 0 0 26 Mar 2025
Mitigating Low-Level Visual Hallucinations Requires Self-Awareness: Database, Model and Training Strategy Yinan Sun Xiongkuo Min Zicheng Zhang Yixuan Gao Yuhang Cao Guangtao Zhai VLM 64 0 0 26 Mar 2025
Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization Zefeng Zhang Hengzhu Tang Shuaiyi Nie Zhenyu Zhang Yiming Ren Zhenyang Li Dawei Yin Duohe Ma Tingwen Liu 55 0 0 23 Mar 2025
DeepMesh: Auto-Regressive Artist-mesh Creation with Reinforcement Learning R. Zhao Junliang Ye Zhilin Wang Guangce Liu Yiwen Chen Yikai Wang Jun Zhu AI4CE 47 0 0 19 Mar 2025
Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations Shuo Li Jiajun Sun Guodong Zheng Xiaoran Fan Yujiong Shen ... Wenming Tan Tao Ji Tao Gui Qi Zhang Xuanjing Huang AAML VLM 92 1 0 19 Mar 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuyao Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang Tieniu Tan 221 2 0 18 Mar 2025
Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song H. Shen 75 1 0 11 Mar 2025
Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning Qinghao Ye Xianhan Zeng Fu Li Chong Li Haoqi Fan CoGe 86 2 0 10 Mar 2025
SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner Kejia Chen Jiawen Zhang Jiacong Hu Jiazhen Yang Jian Lou Zunlei Feng Mingli Song 71 0 0 06 Mar 2025
Visual-RFT: Visual Reinforcement Fine-Tuning Ziyu Liu Zeyi Sun Yuhang Zang Xiaoyi Dong Yuhang Cao Haodong Duan Dahua Lin Jiaqi Wang ObjD VLM LRM 75 48 0 03 Mar 2025
HalCECE: A Framework for Explainable Hallucination Detection through Conceptual Counterfactuals in Image Captioning Maria Lymperaiou Giorgos Filandrianos Angeliki Dimitriou Athanasios Voulodimos Giorgos Stamou MLLM 40 0 0 01 Mar 2025
Octopus: Alleviating Hallucination via Dynamic Contrastive Decoding Wei Suo Lijun Zhang Mengyang Sun Lin Yuanbo Wu Peng Wang Yuyao Zhang MLLM VLM 47 1 0 01 Mar 2025
Towards Statistical Factuality Guarantee for Large Vision-Language Models Zechao Li Chao Yan Nicholas J. Jackson Wendi Cui B. Li Jiaxin Zhang Bradley Malin 73 0 0 27 Feb 2025
Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models Yaqi Sun Kyohei Atarashi Koh Takeuchi Hisashi Kashima MLLM 56 0 0 24 Feb 2025
Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding Kyungmin Min Minbeom Kim Kang-il Lee Dongryeol Lee Kyomin Jung MLLM 92 4 0 20 Feb 2025
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Model Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Ziyu Liu ... Haodong Duan Feiyu Xiong Kai Chen Dahua Lin Jiaqi Wang VLM 80 18 0 21 Jan 2025
Feedback-Driven Vision-Language Alignment with Minimal Human Supervision Giorgio Giannone Ruoteng Li Qianli Feng Evgeny Perevodchikov Rui Chen Aleix M. Martinez VLM 69 0 0 08 Jan 2025
Multimodal Preference Data Synthetic Alignment with Reward Model Robert Wijaya Ngoc-Bao Nguyen Ngai-man Cheung MLLM SyDa 62 2 0 23 Dec 2024
Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence Jinghan He Kuan Zhu Haiyun Guo Sihang Li Zhenglin Hua Yuheng Jia Ming Tang Tat-Seng Chua Jinqiao Wang VLM 95 5 0 18 Dec 2024
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization Kangyu Zhu Peng Xia Yun Li Hongtu Zhu Sheng Wang Huaxiu Yao 111 1 0 09 Dec 2024
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan S. Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 104 15 0 03 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 108 1 0 03 Dec 2024
Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approach Shijian Deng Wentian Zhao Yu-Jhe Li Kun Wan Daniel Miranda Ajinkya Kale Yapeng Tian LRM 82 6 0 26 Nov 2024
Video-Text Dataset Construction from Multi-AI Feedback: Promoting Weak-to-Strong Preference Learning for Video Large Language Models Hao Yi Qingyang Li Yihan Hu Fuzheng Zhang Di Zhang Yong Liu VGen 81 0 0 25 Nov 2024
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models Junzhe Chen Tianshu Zhang S. Huang Yuwei Niu Linfeng Zhang Lijie Wen Xuming Hu MLLM VLM 257 2 0 22 Nov 2024
Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization Yuhan Fu Ruobing Xie Xingchen Sun Zhanhui Kang Xirong Li MLLM 40 4 0 15 Nov 2024
V-DPO: Mitigating Hallucination in Large Vision Language Models via Vision-Guided Direct Preference Optimization Yuxi Xie Guanzhen Li Xiao Xu Min-Yen Kan MLLM VLM 60 16 0 05 Nov 2024
Unified Triplet-Level Hallucination Evaluation for Large Vision-Language Models J. Wu Tsz Ting Chung Kai Chen Dit-Yan Yeung VLM LRM 67 3 0 30 Oct 2024
Vision-Language Models Can Self-Improve Reasoning via Reflection Kanzhi Cheng Yantao Li Fangzhi Xu Jianbing Zhang Hao Zhou Yang Liu ReLM LRM 55 18 0 30 Oct 2024
MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models Ziyu Liu Yuhang Zang Xiaoyi Dong Pan Zhang Yuhang Cao Haodong Duan Zeang Sheng Yuanjun Xiong Dahua Lin Jiaqi Wang 42 7 0 23 Oct 2024
Reducing Hallucinations in Vision-Language Models via Latent Space Steering Sheng Liu Haotian Ye Lei Xing James Zou VLM LLMSV 55 5 0 21 Oct 2024
A Survey of Hallucination in Large Visual Language Models Wei Lan Wenyi Chen Qingfeng Chen Shirui Pan Huiyu Zhou Yi-Lun Pan LRM 33 4 0 20 Oct 2024
Modality-Fair Preference Optimization for Trustworthy MLLM Alignment Songtao Jiang Yan Zhang Ruizhe Chen Yeying Jin Zuozhu Liu MLLM MoE 34 6 0 20 Oct 2024
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment Chenhang Cui An Zhang Yiyang Zhou Zhaorun Chen Gelei Deng Huaxiu Yao Tat-Seng Chua 73 4 0 18 Oct 2024
Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs Shuo Li Tao Ji Xiaoran Fan Linsheng Lu L. Yang ... Yansen Wang Xiaohui Zhao Tao Gui Qi Zhang Xuanjing Huang 52 0 0 15 Oct 2024
VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang L. Chen Yazheng Yang Benyou Wang Lingpeng Kong Qiang Liu VLM ALM 36 17 0 12 Oct 2024
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models Qin Liu Chao Shang Ling Liu Nikolaos Pappas Jie Ma Neha Anna John Srikanth Doss Kadarundalagi Raghuram Doss Lluís Marquez Miguel Ballesteros Yassine Benajiba 44 4 0 11 Oct 2024
From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models Yuying Shang Xinyi Zeng Yutao Zhu Xiao Yang Zhengwei Fang Jingyuan Zhang Jiawei Chen Zinan Liu Yu Tian VLM MLLM 191 1 0 09 Oct 2024
Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models Yufang Liu Tao Ji Changzhi Sun Yuanbin Wu Aimin Zhou VLM MLLM 49 2 0 04 Oct 2024
HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding Fan Yuan Chi Qin Xiaogang Xu Piji Li VLM MLLM 33 4 0 30 Sep 2024
Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding Xiaoyu Liang Jiayuan Yu Lianrui Mu Jiedong Zhuang Jiaqi Hu Yuchen Yang Jiangnan Ye Lu Lu Jian Chen Haoji Hu VLM 45 2 0 10 Sep 2024
LLaVA-MoD: Making LLaVA Tiny via MoE Knowledge Distillation Fangxun Shu Yue Liao Le Zhuo Chenning Xu Guanghao Zhang ... Bolin Li Zhelun Yu Si Liu Hongsheng Li Hao Jiang VLM MoE 32 9 0 28 Aug 2024