Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites

4 December 2023

ArXiv (abs)PDF HTML Github (8★)

Papers citing "Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites"

20 / 20 papers shown

Title
MetaToken: Detecting Hallucination in Image Descriptions by Meta Classification Laura Fieback Jakob Spiegelberg Hanno Gottschalk MLLM 189 5 0 29 May 2024
Hallucination of Multimodal Large Language Models: A Survey Zechen Bai Pichao Wang Tianjun Xiao Tong He Zongbo Han Zheng Zhang Mike Zheng Shou VLM LRM 204 195 0 29 Apr 2024
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 136 2,095 0 11 May 2023
MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action Zhengyuan Yang Linjie Li Jianfeng Wang Kevin Qinghong Lin E. Azarnasab Faisal Ahmed Zicheng Liu Ce Liu Michael Zeng Lijuan Wang ReLM KELM LRM 105 394 0 20 Mar 2023
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning Zhiyang Xu Ying Shen Lifu Huang MLLM 103 120 0 21 Dec 2022
Scaling Instruction-Finetuned Language Models Hyung Won Chung Le Hou Shayne Longpre Barret Zoph Yi Tay ... Jacob Devlin Adam Roberts Denny Zhou Quoc V. Le Jason W. Wei ReLM LRM 208 3,150 0 20 Oct 2022
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training Wenliang Dai Zihan Liu Ziwei Ji Dan Su Pascale Fung MLLM VLM 82 67 0 14 Oct 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 288 1,296 0 20 Sep 2022
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge Dustin Schwenk Apoorv Khandelwal Christopher Clark Kenneth Marino Roozbeh Mottaghi 69 551 0 03 Jun 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 555 4,409 0 28 Jan 2022
CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning Xiangru Tang Arjun Nair Borui Wang Bingyao Wang Jai Desai Aaron Wade Haoran Li Asli Celikyilmaz Yashar Mehdad Dragomir R. Radev HILM 43 63 0 16 Dec 2021
Let there be a clock on the beach: Reducing Object Hallucination in Image Captioning Ali Furkan Biten L. G. I. Bigorda Dimosthenis Karatzas 139 63 0 04 Oct 2021
CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization Shuyang Cao Lu Wang HILM 64 181 0 19 Sep 2021
The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey Yi-Chong Huang Xiachong Feng Xiaocheng Feng Bing Qin HILM 173 108 0 30 Apr 2021
Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection Sihao Chen Fan Zhang Kazoo Sone Dan Roth HILM 85 107 0 19 Apr 2021
Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding Nouha Dziri Andrea Madotto Osmar Zaiane A. Bose HILM 76 135 0 17 Apr 2021
Retrieval Augmentation Reduces Hallucination in Conversation Kurt Shuster Spencer Poff Moya Chen Douwe Kiela Jason Weston HILM 95 742 0 15 Apr 2021
On Hallucination and Predictive Uncertainty in Conditional Language Generation Yijun Xiao Wenjie Wang HILM 156 192 0 28 Mar 2021
A Controllable Model of Grounded Response Generation Zeqiu Wu Michel Galley Chris Brockett Yizhe Zhang Xiang Gao ... Rik Koncel-Kedziorski Jianfeng Gao Hannaneh Hajishirzi Mari Ostendorf Bill Dolan 61 86 0 01 May 2020
nocaps: novel object captioning at scale Harsh Agrawal Karan Desai Yufei Wang Xinlei Chen Rishabh Jain Mark Johnson Dhruv Batra Devi Parikh Stefan Lee Peter Anderson VLM 131 486 0 20 Dec 2018