Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion

6 February 2025

Papers citing "Cross the Gap: Exposing the Intra-modal Misalignment in CLIP via Modality Inversion"

43 / 43 papers shown

Title
Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models Weihao Xuan Qingcheng Zeng Heli Qi Junjue Wang Naoto Yokoya 46 0 0 26 May 2025
Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting Dominic Maggio Luca Carlone 412 0 0 07 Mar 2025
Towards flexible perception with visual memory Robert Geirhos P. Jaini Austin Stone Sourabh Medapati Xi Yi G. Toderici Abhijit Ogale Jonathon Shlens 62 1 0 15 Aug 2024
iSEARLE: Improving Textual Inversion for Zero-Shot Composed Image Retrieval Lorenzo Agnolucci Alberto Baldrati Marco Bertini A. Bimbo 83 14 0 05 May 2024
Leveraging Cross-Modal Neighbor Representation for Improved CLIP Classification Chao Yi Lu Ren De-Chuan Zhan Han-Jia Ye CLIP VLM 39 6 0 27 Apr 2024
Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models Simon Schrodi David T. Hoffmann Max Argus Volker Fischer Thomas Brox VLM 113 4 0 11 Apr 2024
ECLIPSE: A Resource-Efficient Text-to-Image Prior for Image Generations Maitreya Patel Changhoon Kim Sheng Cheng Chitta Baral Yezhou Yang VLM 53 19 0 07 Dec 2023
AVID: Any-Length Video Inpainting with Diffusion Model Zhixing Zhang Bichen Wu Xiaoyan Wang Yaqiao Luo Luxin Zhang Yinan Zhao Peter Vajda Dimitris N. Metaxas Licheng Yu VGen DiffM 81 42 0 06 Dec 2023
DataComp: In search of the next generation of multimodal datasets S. Gadre Gabriel Ilharco Alex Fang J. Hayase Georgios Smyrnis ... A. Dimakis J. Jitsev Y. Carmon Vaishaal Shankar Ludwig Schmidt VLM 94 448 0 27 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 569 4,910 0 17 Apr 2023
CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes Maria Parelli Alexandros Delitzas Nikolas Hars G. Vlassis Sotiris Anagnostidis Gregor Bachmann Thomas Hofmann CLIP 62 50 0 12 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 237 1,200 0 27 Mar 2023
Zero-Shot Composed Image Retrieval with Textual Inversion Alberto Baldrati Lorenzo Agnolucci Marco Bertini A. Bimbo 79 114 0 27 Mar 2023
DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training Wei Li Linchao Zhu Longyin Wen Yi Yang VLM 87 89 0 06 Mar 2023
Diagnosing and Rectifying Vision Models using Language Yuhui Zhang Jeff Z. HaoChen Shih-Cheng Huang Kuan-Chieh Wang James Zou Serena Yeung 79 48 0 08 Feb 2023
Structure and Content-Guided Video Synthesis with Diffusion Models Patrick Esser Johnathan Chiu Parmida Atighehchian Jonathan Granskog Anastasis Germanidis DiffM VGen 172 537 0 06 Feb 2023
SuS-X: Training-Free Name-Only Transfer of Vision-Language Models Vishaal Udandarao Ankush Gupta Samuel Albanie VLM MLLM 78 103 0 28 Nov 2022
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 200 3,493 0 16 Oct 2022
DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation Nataniel Ruiz Yuanzhen Li Varun Jampani Yael Pritch Michael Rubinstein Kfir Aberman 279 2,888 0 25 Aug 2022
An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion Rinon Gal Yuval Alaluf Yuval Atzmon Or Patashnik Amit H. Bermano Gal Chechik Daniel Cohen-Or 164 1,894 0 02 Aug 2022
CoCa: Contrastive Captioners are Image-Text Foundation Models Jiahui Yu Zirui Wang Vijay Vasudevan Legg Yeung Mojtaba Seyedhosseini Yonghui Wu VLM CLIP OffRL 167 1,307 0 04 May 2022
Hierarchical Text-Conditional Image Generation with CLIP Latents Aditya A. Ramesh Prafulla Dhariwal Alex Nichol Casey Chu Mark Chen VLM DiffM 413 6,908 0 13 Apr 2022
CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment Haoyu Song Li Dong Weinan Zhang Ting Liu Furu Wei VLM CLIP 81 139 0 14 Mar 2022
Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning Weixin Liang Yuhui Zhang Yongchan Kwon Serena Yeung James Zou VLM 123 428 0 03 Mar 2022
SLIP: Self-supervision meets Language-Image Pre-training Norman Mu Alexander Kirillov David Wagner Saining Xie VLM CLIP 143 490 0 23 Dec 2021
Image Segmentation Using Text and Image Prompts Timo Lüddecke Alexander S. Ecker CLIP VLM 144 472 0 18 Dec 2021
Extract Free Dense Labels from CLIP Chong Zhou Chen Change Loy Bo Dai VLM CLIP 155 481 0 02 Dec 2021
LiT: Zero-Shot Transfer with Locked-image text Tuning Xiaohua Zhai Tianlin Li Basil Mustafa Andreas Steiner Daniel Keysers Alexander Kolesnikov Lucas Beyer VLM 108 560 0 15 Nov 2021
Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm Yangguang Li Feng Liang Lichen Zhao Yufeng Cui Wanli Ouyang Jing Shao F. Yu Junjie Yan VLM CLIP 150 458 0 11 Oct 2021
Learning to Prompt for Vision-Language Models Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu VPVLM CLIP VLM 505 2,409 0 02 Sep 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 967 29,810 0 26 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 459 3,893 0 11 Feb 2021
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale Alexey Dosovitskiy Lucas Beyer Alexander Kolesnikov Dirk Weissenborn Xiaohua Zhai ... Matthias Minderer G. Heigold Sylvain Gelly Jakob Uszkoreit N. Houlsby ViT 670 41,430 0 22 Oct 2020
A Simple Framework for Contrastive Learning of Visual Representations Ting-Li Chen Simon Kornblith Mohammad Norouzi Geoffrey E. Hinton SSL 375 18,866 0 13 Feb 2020
nocaps: novel object captioning at scale Harsh Agrawal Karan Desai Yufei Wang Xinlei Chen Rishabh Jain Mark Johnson Dhruv Batra Devi Parikh Stefan Lee Peter Anderson VLM 131 486 0 20 Dec 2018
Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking Filip Radenovic Ahmet Iscen Giorgos Tolias Yannis Avrithis Ondřej Chum 55 380 0 29 Mar 2018
EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification P. Helber B. Bischke Andreas Dengel Damian Borth 154 1,830 0 31 Aug 2017
Deep Metric Learning via Lifted Structured Feature Embedding Hyun Oh Song Yu Xiang Stefanie Jegelka Silvio Savarese FedML SSL DML 97 1,643 0 19 Nov 2015
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models Bryan A. Plummer Liwei Wang Christopher M. Cervantes Juan C. Caicedo Julia Hockenmaier Svetlana Lazebnik 208 2,072 0 19 May 2015
Deep Visual-Semantic Alignments for Generating Image Descriptions A. Karpathy Li Fei-Fei 144 5,591 0 07 Dec 2014
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 424 43,814 0 01 May 2014
Fine-Grained Visual Classification of Aircraft Subhransu Maji Esa Rahtu Arno Solin Matthew Blaschko Andrea Vedaldi 126 2,271 0 21 Jun 2013
UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild K. Soomro Amir Zamir M. Shah CLIP VGen 160 6,164 0 03 Dec 2012