Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

4 March 2024

David Wan

Jaemin Cho

Elias Stengel-Eskin

Mohit Bansal

VLM

ObjD

ArXiv (abs)PDF HTML

Papers citing "Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training"

50 / 58 papers shown

Title
Policy Contrastive Decoding for Robotic Foundation Models Shihan Wu Ji Zhang Xu Luo Junlin Xie Jingkuan Song Heng Tao Shen Lianli Gao OffRL 243 0 0 19 May 2025
Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation Ziqiao Ma Jing Ding Xuejun Zhang Dezhi Luo Jiahe Ding Sihan Xu Yuchen Huang Run Peng Joyce Chai 206 0 0 22 Apr 2025
Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding Kyungmin Min Minbeom Kim Kang-il Lee Dongryeol Lee Kyomin Jung MLLM 146 7 0 20 Feb 2025
Towards Visual Grounding: A Survey Linhui Xiao Xiaoshan Yang X. Lan Yaowei Wang Changsheng Xu ObjD 230 5 0 31 Dec 2024
VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding Jiaqi Wang Yifei Gao Jitao Sang MLLM 165 2 0 24 Nov 2024
AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledge Han Wang Archiki Prasad Elias Stengel-Eskin Joey Tianyi Zhou 123 9 0 11 Sep 2024
Contextual Emotion Recognition using Large Vision Language Models Yasaman Etesam Özge Nilay Yalçin Chuxuan Zhang Angelica Lim VLM 99 4 0 14 May 2024
Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance Linxi Zhao Yihe Deng Weitong Zhang Q. Gu MLLM 72 1 0 13 Feb 2024
ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts Mu Cai Haotian Liu Dennis Park Siva Karthik Mustikovela Gregory P. Meyer Yuning Chai Yong Jae Lee VLM LRM MLLM 84 99 0 01 Dec 2023
Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding Sicong Leng Hang Zhang Guanzheng Chen Xin Li Shijian Lu Chunyan Miao Li Bing VLM MLLM 149 239 0 28 Nov 2023
What's "up" with vision-language models? Investigating their struggle with spatial reasoning Amita Kamath Jack Hessel Kai-Wei Chang LRM CoGe 63 117 0 30 Oct 2023
Contrastive Decoding Improves Reasoning in Large Language Models Sean O'Brien Mike Lewis SyDa LRM ReLM 75 38 0 17 Sep 2023
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond Jinze Bai Shuai Bai Shusheng Yang Shijie Wang Sinan Tan Peng Wang Junyang Lin Chang Zhou Jingren Zhou MLLM VLM ObjD 133 931 0 24 Aug 2023
Guiding Image Captioning Models Toward More Specific Captions Simon Kornblith Lala Li Zirui Wang Thao Nguyen 91 15 0 31 Jul 2023
GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interest Shilong Zhang Pei Sun Shoufa Chen Min Xiao Wenqi Shao Wenwei Zhang Yu Liu Kai-xiang Chen Ping Luo MLLM VLM 146 237 0 07 Jul 2023
Stay on topic with Classifier-Free Guidance Guillaume Sanchez Honglu Fan Alexander Spangher Elad Levi Pawan Sasanka Ammanamanchi Stella Biderman 3DV 82 55 0 30 Jun 2023
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic Ke Chen Zhao Zhang Weili Zeng Richong Zhang Feng Zhu Rui Zhao ObjD 95 650 0 27 Jun 2023
SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality Cheng-Yu Hsieh Jieyu Zhang Zixian Ma Aniruddha Kembhavi Ranjay Krishna CoGe 106 131 0 26 Jun 2023
PaLI-X: On Scaling up a Multilingual Vision and Language Model Xi Chen Josip Djolonga Piotr Padlewski Basil Mustafa Soravit Changpinyo ... Mojtaba Seyedhosseini A. Angelova Xiaohua Zhai N. Houlsby Radu Soricut VLM 130 202 0 29 May 2023
Trusting Your Evidence: Hallucinate Less with Context-aware Decoding Weijia Shi Xiaochuang Han M. Lewis Yulia Tsvetkov Luke Zettlemoyer Scott Yih HILM 63 212 0 24 May 2023
What You See is What You Read? Improving Text-Image Alignment Evaluation Michal Yarom Yonatan Bitton Soravit Changpinyo Roee Aharoni Jonathan Herzig Oran Lang E. Ofek Idan Szpektor EGVM 97 85 0 17 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 134 2,095 0 11 May 2023
COLA: A Benchmark for Compositional Text-to-image Retrieval Arijit Ray Filip Radenovic Abhimanyu Dubey Bryan A. Plummer Ranjay Krishna Kate Saenko CoGe VLM 74 38 0 05 May 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 567 4,910 0 17 Apr 2023
Segment Everything Everywhere All at Once Xueyan Zou Jianwei Yang Hao Zhang Feng Li Linjie Li Jianfeng Wang Lijuan Wang Jianfeng Gao Yong Jae Lee MLLM VLM 89 488 0 13 Apr 2023
What does CLIP know about a red circle? Visual prompt engineering for VLMs Aleksandar Shtedritski Christian Rupprecht Andrea Vedaldi VLM MLLM 100 160 0 13 Apr 2023
Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection Shilong Liu Zhaoyang Zeng Tianhe Ren Feng Li Hao Zhang ... Chun-yue Li Jianwei Yang Hang Su Jun Zhu Lei Zhang ObjD 189 2,015 0 09 Mar 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 429 4,641 0 30 Jan 2023
Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpainting Su Wang Chitwan Saharia Ceslee Montgomery Jordi Pont-Tuset Shai Noy ... Radu Soricut Jason Baldridge Mohammad Norouzi Peter Anderson William Chan 74 187 0 13 Dec 2022
CREPE: Can Vision-Language Foundation Models Reason Compositionally? Zixian Ma Jerry Hong Mustafa Omer Gul Mona Gandhi Irena Gao Ranjay Krishna CoGe 79 141 0 13 Dec 2022
Contrastive Decoding: Open-ended Text Generation as Optimization Xiang Lisa Li Ari Holtzman Daniel Fried Percy Liang Jason Eisner Tatsunori Hashimoto Luke Zettlemoyer M. Lewis 115 372 0 27 Oct 2022
MaPLe: Multi-modal Prompt Learning Muhammad Uzair Khattak H. Rasheed Muhammad Maaz Salman Khan Fahad Shahbaz Khan VPVLM VLM 256 568 0 06 Oct 2022
Visual Prompting via Image Inpainting Amir Bar Yossi Gandelsman Trevor Darrell Amir Globerson Alexei A. Efros VLM VPVLM 60 211 0 01 Sep 2022
Generative Bias for Robust Visual Question Answering Jae-Won Cho Dong-Jin Kim H. Ryu In So Kweon OOD CML 71 20 0 01 Aug 2022
Classifier-Free Diffusion Guidance Jonathan Ho Tim Salimans FaML 196 3,963 0 26 Jul 2022
VisFIS: Visual Feature Importance Supervision with Right-for-the-Right-Reason Objectives Zhuofan Ying Peter Hase Joey Tianyi Zhou LRM 66 13 0 22 Jun 2022
GLIPv2: Unifying Localization and Vision-Language Understanding Haotian Zhang Pengchuan Zhang Xiaowei Hu Yen-Chun Chen Liunian Harold Li Xiyang Dai Lijuan Wang Lu Yuan Lei Li Jianfeng Gao ObjD VLM 90 300 0 12 Jun 2022
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding Chitwan Saharia William Chan Saurabh Saxena Lala Li Jay Whang ... Raphael Gontijo-Lopes Tim Salimans Jonathan Ho David J Fleet Mohammad Norouzi VLM 461 6,067 0 23 May 2022
Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality Tristan Thrush Ryan Jiang Max Bartolo Amanpreet Singh Adina Williams Douwe Kiela Candace Ross CoGe 106 427 0 07 Apr 2022
FocalClick: Towards Practical Interactive Image Segmentation Xi Chen Zhiyan Zhao Yilei Zhang Manni Duan Donglian Qi Hengshuang Zhao 54 129 0 06 Apr 2022
Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priors Oran Gafni Adam Polyak Oron Ashual Shelly Sheynin Devi Parikh Yaniv Taigman DiffM 81 524 0 24 Mar 2022
FLAVA: A Foundational Language And Vision Alignment Model Amanpreet Singh Ronghang Hu Vedanuj Goswami Guillaume Couairon Wojciech Galuba Marcus Rohrbach Douwe Kiela CLIP VLM 104 715 0 08 Dec 2021
Grounded Language-Image Pre-training Liunian Harold Li Pengchuan Zhang Haotian Zhang Jianwei Yang Chunyuan Li ... Lu Yuan Lei Zhang Lei Li Kai-Wei Chang Jianfeng Gao ObjD VLM 129 1,066 0 07 Dec 2021
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models Yuan Yao Ao Zhang Zhengyan Zhang Zhiyuan Liu Tat-Seng Chua Maosong Sun MLLM VPVLM VLM 281 224 0 24 Sep 2021
MERLOT: Multimodal Neural Script Knowledge Models Rowan Zellers Ximing Lu Jack Hessel Youngjae Yu J. S. Park Jize Cao Ali Farhadi Yejin Choi VLM LRM 104 383 0 04 Jun 2021
MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding Aishwarya Kamath Mannat Singh Yann LeCun Gabriel Synnaeve Ishan Misra Nicolas Carion ObjD VLM 182 889 0 26 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 967 29,810 0 26 Feb 2021
Answer Questions with Right Image Regions: A Visual Attention Regularization Approach Yebin Liu Yangyang Guo Jianhua Yin Xuemeng Song Weifeng Liu Liqiang Nie 70 29 0 03 Feb 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 670 41,430 0 22 Oct 2020
Self-Critical Reasoning for Robust Visual Question Answering Jialin Wu Raymond J. Mooney OOD NAI 73 161 0 24 May 2019