What does CLIP know about a red circle? Visual prompt engineering for VLMs

13 April 2023

Aleksandar Shtedritski

Christian Rupprecht

Andrea Vedaldi

VLM

MLLM

ArXiv PDF HTML

Papers citing "What does CLIP know about a red circle? Visual prompt engineering for VLMs"

37 / 37 papers shown

Title
Show or Tell? A Benchmark To Evaluate Visual and Textual Prompts in Semantic Segmentation Gabriele Rosi Fabio Cermelli VLM 42 0 0 06 May 2025
RESAnything: Attribute Prompting for Arbitrary Referring Segmentation Ruiqi Wang Hao Zhang VLM 56 0 0 03 May 2025
Black-Box Visual Prompt Engineering for Mitigating Object Hallucination in Large Vision Language Models Sangmin Woo Kang Zhou Yun Zhou Shuai Wang Sheng Guan Haibo Ding Lin Lee Cheong VPVLM 83 0 0 30 Apr 2025
Vision-Language Models Are Not Pragmatically Competent in Referring Expression Generation Ziqiao Ma Jing Ding Xuejun Zhang Dezhi Luo Jiahe Ding Sihan Xu Yuchen Huang Run Peng Joyce Chai 51 0 0 22 Apr 2025
Post-Hurricane Debris Segmentation Using Fine-Tuned Foundational Vision Models Kooshan Amini Yuhao Liu Jamie Ellen Padgett Guha Balakrishnan Ashok Veeraraghavan 33 0 0 17 Apr 2025
Aligning Generative Denoising with Discriminative Objectives Unleashes Diffusion for Visual Perception Ziqi Pang Xin Xu Yu-Xiong Wang DiffM 65 0 0 15 Apr 2025
How Can Objects Help Video-Language Understanding? Zitian Tang Shijie Wang Junho Cho Jaewook Yoo Chen Sun 42 0 0 10 Apr 2025
4D LangSplat: 4D Language Gaussian Splatting via Multimodal Large Language Models Wanhua Li Renping Zhou Jiawei Zhou Yingwei Song Johannes Herter Minghan Qin Gao Huang Hanspeter Pfister 3DGS VLM 66 0 0 13 Mar 2025
Show and Tell: Visually Explainable Deep Neural Nets via Spatially-Aware Concept Bottleneck Models Itay Benou Tammy Riklin-Raviv 67 0 0 27 Feb 2025
A Comprehensive Survey of Foundation Models in Medicine Wasif Khan Seowung Leem Kyle B. See Joshua K. Wong Shaoting Zhang R. Fang AI4CE LM&MA VLM 105 18 0 17 Jan 2025
VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks Jiannan Wu Muyan Zhong Sen Xing Zeqiang Lai Zhaoyang Liu ... Lewei Lu Tong Lu Ping Luo Yu Qiao Jifeng Dai MLLM VLM LRM 99 48 0 03 Jan 2025
Differentiable Prompt Learning for Vision Language Models Zhenhan Huang Tejaswini Pedapati Pin-Yu Chen Jianxi Gao VLM 28 0 0 03 Jan 2025
MVREC: A General Few-shot Defect Classification Model Using Multi-View Region-Context Shuai Lyu Fangjian Liao Zeqi Ma Rongchen Zhang Dongmei Mo W. Wong 76 1 0 22 Dec 2024
Driving by the Rules: A Benchmark for Integrating Traffic Sign Regulations into Vectorized HD Map Xinyuan Chang Maixuan Xue Xinran Liu Zheng Pan Xing Wei 48 1 0 31 Oct 2024
The Labyrinth of Links: Navigating the Associative Maze of Multi-modal LLMs Hong Li Nanxi Li Yuanjie Chen Jianbin Zhu Qinlu Guo Cewu Lu Yong-Lu Li MLLM 34 1 0 02 Oct 2024
VL-TGS: Trajectory Generation and Selection using Vision Language Models in Mapless Outdoor Environments Daeun Song Jing Liang Xuesu Xiao Dinesh Manocha 48 4 0 05 Aug 2024
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Ming-Kuan Wu Xinyue Cai Jiayi Ji Jiale Li Oucheng Huang Gen Luo Hao Fei Xiaoshuai Sun Rongrong Ji MLLM 45 7 0 31 Jul 2024
Learning Visual Grounding from Generative Vision and Language Model Shijie Wang Dahun Kim A. Taalimi Chen Sun Weicheng Kuo ObjD 34 5 0 18 Jul 2024
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Xiang Li Cristina Mata J. Park Kumara Kahatapitiya Yoo Sung Jang ... Kanchana Ranasinghe R. Burgert Mu Cai Yong Jae Lee Michael S. Ryoo LM&Ro 64 25 0 28 Jun 2024
VANE-Bench: Video Anomaly Evaluation Benchmark for Conversational LMMs Rohit K Bharadwaj Hanan Gani Muzammal Naseer F. Khan Salman Khan 53 3 0 14 Jun 2024
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs An Yan Zhengyuan Yang Junda Wu Wanrong Zhu Jianwei Yang ... K. Lin Jianfeng Wang Julian McAuley Jianfeng Gao Lijuan Wang LRM 34 12 0 25 Apr 2024
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want Weifeng Lin Xinyu Wei Ruichuan An Peng Gao Bocheng Zou Yulin Luo Siyuan Huang Shanghang Zhang Hongsheng Li VLM 63 33 0 29 Mar 2024
Data-Efficient 3D Visual Grounding via Order-Aware Referring Tung-Yu Wu Sheng-Yu Huang Yu-Chiang Frank Wang 34 0 0 25 Mar 2024
Spurious Feature Eraser: Stabilizing Test-Time Adaptation for Vision-Language Foundation Model Huan Ma Yan Zhu Changqing Zhang Peilin Zhao Baoyuan Wu Long-Kai Huang Qinghua Hu Bing Wu VLM 67 1 0 01 Mar 2024
CoLLaVO: Crayon Large Language and Vision mOdel Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro VLM MLLM 29 16 0 17 Feb 2024
Quilt-LLaVA: Visual Instruction Tuning by Extracting Localized Narratives from Open-Source Histopathology Videos M. S. Seyfioglu Wisdom O. Ikezogwo Fatemeh Ghezloo Ranjay Krishna Linda G. Shapiro 30 33 0 07 Dec 2023
Alpha-CLIP: A CLIP Model Focusing on Wherever You Want Zeyi Sun Ye Fang Tong Wu Pan Zhang Yuhang Zang Shu Kong Yuanjun Xiong Dahua Lin Jiaqi Wang VLM CLIP 37 83 0 06 Dec 2023
ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation Yasheng Sun Yifan Yang Houwen Peng Yifei Shen Yuqing Yang Hang-Rui Hu Lili Qiu Hideki Koike DiffM LM&Ro 37 33 0 02 Aug 2023
Foundational Models Defining a New Era in Vision: A Survey and Outlook Muhammad Awais Muzammal Naseer Salman Khan Rao Muhammad Anwer Hisham Cholakkal M. Shah Ming Yang F. Khan VLM 26 118 0 25 Jul 2023
Visual Instruction Tuning with Polite Flamingo Delong Chen Jianfeng Liu Wenliang Dai Baoyuan Wang MLLM 28 42 0 03 Jul 2023
VisoGender: A dataset for benchmarking gender bias in image-text pronoun resolution S. Hall F. G. Abrantes Hanwen Zhu Grace A. Sodunke Aleksandar Shtedritski Hannah Rose Kirk CoGe 13 39 0 21 Jun 2023
Segment Any Anomaly without Training via Hybrid Prompt Regularization Yunkang Cao Xiaohao Xu Chen Sun Y. Cheng Zongwei Du Liang Gao Weiming Shen VLM 35 70 0 18 May 2023
A Region-Prompted Adapter Tuning for Visual Abductive Reasoning Hao Zhang Yeo Keat Ee Basura Fernando VLM 27 3 0 18 Mar 2023
GroupViT: Semantic Segmentation Emerges from Text Supervision Jiarui Xu Shalini De Mello Sifei Liu Wonmin Byeon Thomas Breuel Jan Kautz X. Wang ViT VLM 189 499 0 22 Feb 2022
CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models Yuan Yao Ao Zhang Zhengyan Zhang Zhiyuan Liu Tat-Seng Chua Maosong Sun MLLM VPVLM VLM 202 221 0 24 Sep 2021
Are Gender-Neutral Queries Really Gender-Neutral? Mitigating Gender Bias in Image Search Jialu Wang Yang Liu X. Wang FaML 157 95 0 12 Sep 2021
Learning to Prompt for Vision-Language Models Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu VPVLM CLIP VLM 325 2,263 0 02 Sep 2021