Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization

24 May 2024

Beitao Chen

Xinyu Lyu

Lianli Gao

Jingkuan Song

Hengtao Shen

MLLM

ArXiv PDF HTML

Papers citing "Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization"

50 / 52 papers shown

Title
Exploring Hallucination of Large Multimodal Models in Video Understanding: Benchmark, Analysis and Mitigation Hongcheng Gao Jiashu Qu Jingyi Tang Baolong Bi Yi Liu Hongyu Chen Li Liang Li Su Qingming Huang MLLM VLM LRM 131 5 0 25 Mar 2025
Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding Shunqi Mao Chaoyi Zhang Weidong Cai MLLM 361 1 0 13 Mar 2025
TruthPrInt: Mitigating LVLM Object Hallucination Via Latent Truthful-Guided Pre-Intervention Jinhao Duan Fei Kong Hao-Ran Cheng James Diffenderfer B. Kailkhura Lichao Sun Xiaofeng Zhu Xiaoshuang Shi Kaidi Xu 380 2 0 13 Mar 2025
Attention Hijackers: Detect and Disentangle Attention Hijacking in LVLMs for Hallucination Mitigation Beitao Chen Xinyu Lyu Lianli Gao Jingkuan Song Jikang Cheng 133 1 0 11 Mar 2025
ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models Junzhe Chen Tianshu Zhang Shijie Huang Yuwei Niu Linfeng Zhang Lijie Wen Xuming Hu MLLM VLM 394 4 0 22 Nov 2024
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment Chenhang Cui An Zhang Yiyang Zhou Zhaorun Chen Gelei Deng Huaxiu Yao Tat-Seng Chua 121 5 0 18 Oct 2024
Any Target Can be Offense: Adversarial Example Generation via Generalized Latent Infection Youheng Sun Shengming Yuan Xuanhan Wang Lianli Gao Jingkuan Song AAML 47 3 0 17 Jul 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 151 175 0 29 Apr 2024
Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding Xintong Wang Jingheng Pan Liang Ding Christian Biemann MLLM 60 66 0 27 Mar 2024
Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination Dingchen Yang Bowen Cao Guang Chen Changjun Jiang 63 11 0 21 Mar 2024
Multi-Modal Hallucination Control by Visual Information Grounding Alessandro Favero Luca Zancato Matthew Trager Siddharth Choudhary Pramuditha Perera Alessandro Achille Ashwin Swaminathan Stefano Soatto MLLM 110 74 0 20 Mar 2024
HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding Zhaorun Chen Zhuokai Zhao Hongyin Luo Huaxiu Yao Bo Li Jiawei Zhou MLLM 74 69 0 01 Mar 2024
IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding Lanyun Zhu Deyi Ji Tianrun Chen Peng Xu Jieping Ye Jun Liu MLLM 94 51 0 28 Feb 2024
Aligning Modalities in Vision Large Language Models via Preference Fine-tuning Yiyang Zhou Chenhang Cui Rafael Rafailov Chelsea Finn Huaxiu Yao VLM MLLM 79 104 0 18 Feb 2024
EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models Shangyu Xing Fei Zhao Zhen Wu Tuo An Weihao Chen Chunhui Li Jianbing Zhang Xinyu Dai MLLM MU 84 5 0 15 Feb 2024
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback M. Steyvers Yuan Yao Haoye Zhang Taiwen He Yifeng Han ... Xinyue Hu Zhiyuan Liu Hai-Tao Zheng Maosong Sun Tat-Seng Chua MLLM VLM 166 210 0 01 Dec 2023
OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation Qidong Huang Xiao-wen Dong Pan Zhang Bin Wang Conghui He Jiaqi Wang Dahua Lin Weiming Zhang Neng H. Yu MLLM 113 193 0 29 Nov 2023
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding Sicong Leng Hang Zhang Guanzheng Chen Xin Li Shijian Lu Chunyan Miao Li Bing VLM MLLM 136 223 0 28 Nov 2023
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization Zhiyuan Zhao Bin Wang Linke Ouyang Xiao-wen Dong Jiaqi Wang Conghui He MLLM VLM 92 124 0 28 Nov 2023
CogVLM: Visual Expert for Pretrained Language Models Weihan Wang Qingsong Lv Wenmeng Yu Wenyi Hong Ji Qi ... Bin Xu Juanzi Li Yuxiao Dong Ming Ding Jie Tang VLM MLLM 69 485 0 06 Nov 2023
Woodpecker: Hallucination Correction for Multimodal Large Language Models Shukang Yin Chaoyou Fu Sirui Zhao Tong Xu Hao Wang Dianbo Sui Yunhang Shen Ke Li Xingguo Sun Enhong Chen VLM MLLM 82 128 0 24 Oct 2023
Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models Holy Lovenia Wenliang Dai Samuel Cahyawijaya Ziwei Ji Pascale Fung MLLM 57 52 0 09 Oct 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 108 2,672 0 05 Oct 2023
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts Pan Lu Hritik Bansal Tony Xia Jiacheng Liu Chun-yue Li Hannaneh Hajishirzi Hao Cheng Kai-Wei Chang Michel Galley Jianfeng Gao LRM MLLM 91 605 0 03 Oct 2023
Analyzing and Mitigating Object Hallucination in Large Vision-Language Models Yiyang Zhou Chenhang Cui Jaehong Yoon Linjun Zhang Zhun Deng Chelsea Finn Mohit Bansal Huaxiu Yao MLLM 101 178 0 01 Oct 2023
The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision) Zhengyuan Yang Linjie Li Kevin Qinghong Lin Jianfeng Wang Chung-Ching Lin Nasim Shakouri Mahmoudabadi Lijuan Wang LM&MA 49 630 0 29 Sep 2023
DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models Yung-Sung Chuang Yujia Xie Hongyin Luo Yoon Kim James R. Glass Pengcheng He HILM 63 154 0 07 Sep 2023
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models Yue Zhang Yafu Li Leyang Cui Deng Cai Lemao Liu ... Longyue Wang Anh Tuan Luu Wei Bi Freda Shi Shuming Shi RALM LRM HILM 82 553 0 03 Sep 2023
Evaluation and Analysis of Hallucination in Large Vision-Language Models Junyan Wang Yi Zhou Guohai Xu Pengcheng Shi Chenlin Zhao ... Mingshi Yan Ji Zhang Jihua Zhu Jitao Sang Haoyu Tang MLLM 61 68 0 29 Aug 2023
Detecting and Preventing Hallucinations in Large Vision Language Models Anisha Gunjal Jihan Yin Erhan Bas MLLM VLM 52 163 0 11 Aug 2023
Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning Fuxiao Liu Kevin Qinghong Lin Linjie Li Jianfeng Wang Yaser Yacoob Lijuan Wang VLM MLLM 89 275 0 26 Jun 2023
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang ... Xiawu Zheng Ke Li Xing Sun Zhenyu Qiu Rongrong Ji ELM MLLM 77 821 0 23 Jun 2023
Direct Preference Optimization: Your Language Model is Secretly a Reward Model Rafael Rafailov Archit Sharma E. Mitchell Stefano Ermon Christopher D. Manning Chelsea Finn ALM 313 3,895 0 29 May 2023
Evaluating Object Hallucination in Large Vision-Language Models Yifan Li Yifan Du Kun Zhou Jinpeng Wang Wayne Xin Zhao Ji-Rong Wen MLLM LRM 269 768 0 17 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 91 2,047 0 11 May 2023
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models Deyao Zhu Jun Chen Xiaoqian Shen Xiang Li Mohamed Elhoseiny VLM MLLM 126 2,007 0 20 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 433 4,678 0 17 Apr 2023
PaLM-E: An Embodied Multimodal Language Model Danny Driess F. Xia Mehdi S. M. Sajjadi Corey Lynch Aakanksha Chowdhery ... Marc Toussaint Klaus Greff Andy Zeng Igor Mordatch Peter R. Florence LM&Ro 83 1,629 0 06 Mar 2023
REPLUG: Retrieval-Augmented Black-Box Language Models Weijia Shi Sewon Min Michihiro Yasunaga Minjoon Seo Rich James M. Lewis Luke Zettlemoyer Wen-tau Yih RALM VLM KELM 118 631 0 30 Jan 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 401 4,527 0 30 Jan 2023
Contrastive Decoding: Open-ended Text Generation as Optimization Xiang Lisa Li Ari Holtzman Daniel Fried Percy Liang Jason Eisner Tatsunori Hashimoto Luke Zettlemoyer M. Lewis 89 358 0 27 Oct 2022
Scaling Laws for Reward Model Overoptimization Leo Gao John Schulman Jacob Hilton ALM 84 542 0 19 Oct 2022
Natural Color Fool: Towards Boosting Black-box Unrestricted Attacks Shengming Yuan Qilong Zhang Lianli Gao Yaya Cheng Jingkuan Song AAML 59 44 0 05 Oct 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 344 3,515 0 29 Apr 2022
Survey of Hallucination in Natural Language Generation Ziwei Ji Nayeon Lee Rita Frieske Tiezheng Yu D. Su ... Delong Chen Wenliang Dai Ho Shu Chan Andrea Madotto Pascale Fung HILM LRM 186 2,356 0 08 Feb 2022
TruthfulQA: Measuring How Models Mimic Human Falsehoods Stephanie C. Lin Jacob Hilton Owain Evans HILM 125 1,881 0 08 Sep 2021
Detecting Hallucinated Content in Conditional Neural Sequence Generation Chunting Zhou Graham Neubig Jiatao Gu Mona T. Diab P. Guzmán Luke Zettlemoyer Marjan Ghazvininejad HILM 81 200 0 05 Nov 2020
The Curious Case of Neural Text Degeneration Ari Holtzman Jan Buys Li Du Maxwell Forbes Yejin Choi 170 3,160 0 22 Apr 2019
Object Hallucination in Image Captioning Anna Rohrbach Lisa Anne Hendricks Kaylee Burns Trevor Darrell Kate Saenko 169 424 0 06 Sep 2018
Hierarchical Neural Story Generation Angela Fan M. Lewis Yann N. Dauphin DiffM 170 1,615 0 13 May 2018