Image Captioners Are Scalable Vision Learners Too

13 June 2023

Papers citing "Image Captioners Are Scalable Vision Learners Too"

50 / 56 papers shown

Title
Breaking the Modality Barrier: Universal Embedding Learning with Multimodal LLMs Tiancheng Gu Kaicheng Yang Ziyong Feng Xingjun Wang Yanzhao Zhang Dingkun Long Yingda Chen Weidong Cai Jiankang Deng VLM 170 2 0 24 Apr 2025
Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction Vaishnavh Nagarajan Chen Henry Wu Charles Ding Aditi Raghunathan 36 0 0 21 Apr 2025
Perception Encoder: The best visual embeddings are not at the output of the network Daniel Bolya Po-Yao (Bernie) Huang Peize Sun Jang Hyun Cho Andrea Madotto ... Shiyu Dong Nikhila Ravi Daniel Li Piotr Dollár Christoph Feichtenhofer ObjD VOS 103 1 0 17 Apr 2025
Can Masked Autoencoders Also Listen to Birds? Lukas Rauch Ilyass Moummad René Heinrich Alexis Joly Bernhard Sick Christoph Scholz 31 0 0 17 Apr 2025
The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer Weixian Lei Jiacong Wang Haochen Wang X. Li Jun Hao Liew Jiashi Feng Zilong Huang 28 2 0 14 Apr 2025
Scaling Language-Free Visual Representation Learning David Fan Shengbang Tong Jiachen Zhu Koustuv Sinha Zhuang Liu ... Michael G. Rabbat Nicolas Ballas Yann LeCun Amir Bar Saining Xie CLIP VLM 64 2 0 01 Apr 2025
A Frustratingly Simple Yet Highly Effective Attack Baseline: Over 90% Success Rate Against the Strong Black-box Models of GPT-4.5/4o/o1 Zhaoyi Li Xiaohan Zhao Dong-Dong Wu Jiacheng Cui Zhiqiang Shen AAML VLM 75 0 0 13 Mar 2025
MASS: Overcoming Language Bias in Image-Text Matching Jiwan Chung Seungwon Lim Sangkyu Lee Youngjae Yu VLM 32 0 0 20 Jan 2025
A Simple Recipe for Contrastively Pre-training Video-First Encoders Beyond 16 Frames Pinelopi Papalampidi Skanda Koppula Shreya Pathak Justin T Chiu Joseph Heyward Viorica Patraucean Jiajun Shen Antoine Miech Andrew Zisserman Aida Nematzdeh VLM 63 24 0 31 Dec 2024
Bringing Multimodality to Amazon Visual Search System Xinliang Zhu Michael Huang Han Ding Jinyu Yang Kelvin Chen ... Son Dinh Tran Benjamin Z. Yao Doug Gray Anuj Bindal Arnab Dhua 74 3 0 17 Dec 2024
Which Viewpoint Shows it Best? Language for Weakly Supervising View Selection in Multi-view Instructional Videos Sagnik Majumder Tushar Nagarajan Ziad Al-Halah Reina Pradhan Kristen Grauman 31 0 0 13 Nov 2024
Classification Done Right for Vision-Language Pre-Training Zilong Huang Qinghao Ye Bingyi Kang Jiashi Feng Haoqi Fan CLIP VLM 50 2 0 05 Nov 2024
NaturalBench: Evaluating Vision-Language Models on Natural Adversarial Samples Baiqi Li Zhiqiu Lin Wenxuan Peng Jean de Dieu Nyandwi Daniel Jiang Zixian Ma Simran Khanuja Ranjay Krishna Graham Neubig Deva Ramanan AAML CoGe VLM 71 21 0 18 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Zou Tatsunori Hashimoto VLM 70 4 0 14 Oct 2024
The Hard Positive Truth about Vision-Language Compositionality Amita Kamath Cheng-Yu Hsieh Kai-Wei Chang Ranjay Krishna CLIP CoGe VLM 30 5 0 26 Sep 2024
Generative Semantic Communication via Textual Prompts: Latency Performance Tradeoffs Mengmeng Ren Li Qiao Long Yang Zhen Gao Jian Chen Mahdi Boloursaz Mashhadi Pei Xiao Rahim Tafazolli Mehdi Bennis VLM 96 5 0 15 Sep 2024
Foundational Models for Pathology and Endoscopy Images: Application for Gastric Inflammation H. Kerdegari Kyle Higgins Dennis Veselkov I. Laponogov I. Poļaka ... Junior Andrea Pescino M. Leja M. Dinis-Ribeiro T. F. Kanonnikoff Kirill Veselkov 35 3 0 26 Jun 2024
A Simple Framework for Open-Vocabulary Zero-Shot Segmentation Thomas Stegmüller Tim Lebailly Nikola Dukic Behzad Bozorgtabar Tinne Tuytelaars Jean-Philippe Thiran VLM 39 1 0 23 Jun 2024
BiVLC: Extending Vision-Language Compositionality Evaluation with Text-to-Image Retrieval Imanol Miranda Ander Salaberria Eneko Agirre Gorka Azkune CoGe 46 1 0 14 Jun 2024
ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs Irene Huang Wei Lin M. Jehanzeb Mirza Jacob A. Hansen Sivan Doveh ... Trevor Darrel Chuang Gan Aude Oliva Rogerio Feris Leonid Karlinsky CoGe LRM 43 7 0 12 Jun 2024
Nomic Embed Vision: Expanding the Latent Space Zach Nussbaum Brandon Duderstadt Andriy Mulyar VLM 33 5 0 06 Jun 2024
No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models Angeline Pouget Lucas Beyer Emanuele Bugliarello Xiao Wang Andreas Steiner Xiao-Qi Zhai Ibrahim M. Alabdulmohsin VLM 33 7 0 22 May 2024
A Philosophical Introduction to Language Models - Part II: The Way Forward Raphael Milliere Cameron Buckner LRM 66 13 0 06 May 2024
Two Effects, One Trigger: On the Modality Gap, Object Bias, and Information Imbalance in Contrastive Vision-Language Models Simon Schrodi David T. Hoffmann Max Argus Volker Fischer Thomas Brox VLM 58 0 0 11 Apr 2024
Hyperbolic Learning with Synthetic Captions for Open-World Detection Fanjie Kong Yanbei Chen Jiarui Cai Davide Modolo VLM ObjD 31 7 0 07 Apr 2024
VLRM: Vision-Language Models act as Reward Models for Image Captioning Maksim Dzabraev Alexander Kunitsyn Andrei Ivaniuta VLM MLLM 31 3 0 02 Apr 2024
LocCa: Visual Pretraining with Location-aware Captioners Bo Wan Michael Tschannen Yongqin Xian Filip Pavetić Ibrahim M. Alabdulmohsin Xiao Wang André Susano Pinto Andreas Steiner Lucas Beyer Xiao-Qi Zhai VLM 51 6 0 28 Mar 2024
Differentially Private Representation Learning via Image Captioning Tom Sander Yaodong Yu Maziar Sanjabi Alain Durmus Yi Ma Kamalika Chaudhuri Chuan Guo 71 3 0 04 Mar 2024
Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning Maurits J. R. Bleeker Mariya Hendriksen Andrew Yates Maarten de Rijke VLM 40 3 0 27 Feb 2024
CLoVe: Encoding Compositional Language in Contrastive Vision-Language Models Santiago Castro Amir Ziai Avneesh Saluja Zhuoning Yuan Rada Mihalcea MLLM CoGe VLM 34 5 0 22 Feb 2024
Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationships Sebastian Koch Narunas Vaskevicius Mirco Colosi Pedro Hermosilla Timo Ropinski 3DPC 33 26 0 19 Feb 2024
Cacophony: An Improved Contrastive Audio-Text Model Ge Zhu Jordan Darefsky Zhiyao Duan AuLLM 46 11 0 10 Feb 2024
A Probabilistic Model behind Self-Supervised Learning Alice Bizeul Bernhard Schölkopf Carl Allen SSL 26 2 0 02 Feb 2024
Exploring scalable medical image encoders beyond text supervision Fernando Pérez-García Harshita Sharma Sam Bond-Taylor Kenza Bouzid Valentina Salvatelli ... Maria T. A. Wetscherek Noel C. F. Codella Stephanie L. Hyland Javier Alvarez-Valle Ozan Oktay LM&MA MedIm 50 9 0 19 Jan 2024
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs Shengbang Tong Zhuang Liu Yuexiang Zhai Yi Ma Yann LeCun Saining Xie VLM MLLM 41 283 0 11 Jan 2024
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers Aleksandar Stanić Sergi Caelles Michael Tschannen LRM VLM 27 9 0 03 Jan 2024
GIVT: Generative Infinite-Vocabulary Transformers Michael Tschannen Cian Eastwood Fabian Mentzer 31 34 0 04 Dec 2023
SPOT: Self-Training with Patch-Order Permutation for Object-Centric Learning with Autoregressive Transformers Ioannis Kakogeorgiou Spyros Gidaris Konstantinos Karantzalos N. Komodakis ViT OCL 17 12 0 01 Dec 2023
Foundational Models in Medical Imaging: A Comprehensive Survey and Future Vision Bobby Azad Reza Azad Sania Eskandari Afshin Bozorgpour A. Kazerouni I. Rekik Dorit Merhof VLM MedIm 98 59 0 28 Oct 2023
SILC: Improving Vision Language Pretraining with Self-Distillation Muhammad Ferjad Naeem Yongqin Xian Xiaohua Zhai Lukas Hoyer Luc Van Gool F. Tombari VLM 26 33 0 20 Oct 2023
PaLI-3 Vision Language Models: Smaller, Faster, Stronger Xi Chen Xiao Wang Lucas Beyer Alexander Kolesnikov Jialin Wu ... Keran Rong Tianli Yu Daniel Keysers Xiao-Qi Zhai Radu Soricut MLLM VLM 41 94 0 13 Oct 2023
Intriguing properties of generative classifiers P. Jaini Kevin Clark Robert Geirhos BDL 25 33 0 28 Sep 2023
Looking at words and points with attention: a benchmark for text-to-shape coherence Andrea Amaduzzi Giuseppe Lisanti Samuele Salti Luigi Di Stefano 18 2 0 14 Sep 2023
Foundational Models Defining a New Era in Vision: A Survey and Outlook Muhammad Awais Muzammal Naseer Salman Khan Rao Muhammad Anwer Hisham Cholakkal M. Shah Ming Yang F. Khan VLM 38 118 0 25 Jul 2023
SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionality Cheng-Yu Hsieh Jieyu Zhang Zixian Ma Aniruddha Kembhavi Ranjay Krishna CoGe 43 115 0 26 Jun 2023
Revisiting the Role of Language Priors in Vision-Language Models Zhiqiu Lin Xinyue Chen Deepak Pathak Pengchuan Zhang Deva Ramanan VLM 23 22 0 02 Jun 2023
Text encoders bottleneck compositionality in contrastive vision-language models Amita Kamath Jack Hessel Kai-Wei Chang CoGe CLIP VLM 30 19 0 24 May 2023
Vision Learners Meet Web Image-Text Pairs Bingchen Zhao Quan Cui Hao Wu Osamu Yoshie Cheng Yang Oisin Mac Aodha VLM 27 5 0 17 Jan 2023
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 169 263 0 07 Oct 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,137 0 28 Jan 2022