What matters when building vision-language models?

3 May 2024

Papers citing "What matters when building vision-language models?"

33 / 133 papers shown

Title
ReMI: A Dataset for Reasoning with Multiple Images Mehran Kazemi Nishanth Dikkala Ankit Anand Petar Dević Ishita Dasgupta ... Bahare Fatemi Pranjal Awasthi Dee Guo Sreenivas Gollapudi Ahmed Qureshi LRM VLM 44 13 0 13 Jun 2024
mOSCAR: A Large-scale Multilingual and Multimodal Document-level Corpus Matthieu Futeral A. Zebaze Pedro Ortiz Suarez Julien Abadji Rémi Lacroix Cordelia Schmid Rachel Bawden Benoît Sagot 39 3 0 13 Jun 2024
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text Qingyun Li Zhe Chen Weiyun Wang Wenhai Wang Shenglong Ye ... Dahua Lin Yu Qiao Botian Shi Conghui He Jifeng Dai VLM OffRL 56 20 0 12 Jun 2024
Needle In A Multimodal Haystack Weiyun Wang Shuibo Zhang Yiming Ren Yuchen Duan Tiantong Li ... Ping Luo Yu Qiao Jifeng Dai Wenqi Shao Wenhai Wang VLM 59 17 0 11 Jun 2024
Merlin: A Vision Language Foundation Model for 3D Computed Tomography Louis Blankemeier Joseph Paul Cohen Ashwin Kumar Dave Van Veen Syed Jamal Safdar Gardezi ... Andrew L. Wentland C. Langlotz Jason Hom S. Gatidis Akshay S. Chaudhari LM&MA MedIm 49 31 0 10 Jun 2024
F-LMM: Grounding Frozen Large Multimodal Models Size Wu Sheng Jin Wenwei Zhang Lumin Xu Wentao Liu Wei Li Chen Change Loy MLLM 80 12 0 09 Jun 2024
Dragonfly: Multi-Resolution Zoom Supercharges Large Visual-Language Model Kezhen Chen Rahul Thapa Rahul Chalamala Ben Athiwaratkun S. Song James Y. Zou VLM 50 0 0 03 Jun 2024
Visual Perception by Large Language Model's Weights Feipeng Ma Hongwei Xue Guangting Wang Yizhou Zhou Fengyun Rao Shilin Yan Yueyi Zhang Siying Wu Mike Zheng Shou Xiaoyan Sun VLM 25 5 0 30 May 2024
Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare Hanwei Zhu Haoning Wu Yixuan Li Zicheng Zhang Baoliang Chen Lingyu Zhu Yuming Fang Guangtao Zhai Weisi Lin Shiqi Wang 45 18 0 29 May 2024
Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs Jialiang Xu Michael Moor J. Leskovec 32 2 0 29 May 2024
Why are Visually-Grounded Language Models Bad at Image Classification? Yuhui Zhang Alyssa Unell Xiaohan Wang Dhruba Ghosh Yuchang Su Ludwig Schmidt Serena Yeung-Levy VLM 35 27 0 28 May 2024
The Evolution of Multimodal Model Architectures S. Wadekar Abhishek Chaurasia Aman Chadha Eugenio Culurciello 43 14 0 28 May 2024
CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts Jiachen Li Xinyao Wang Sijie Zhu Chia-Wen Kuo Lu Xu Fan Chen Jitesh Jain Humphrey Shi Longyin Wen MLLM MoE 46 26 0 09 May 2024
MANTIS: Interleaved Multi-Image Instruction Tuning Dongfu Jiang Xuan He Huaye Zeng Cong Wei Max W.F. Ku Qian Liu Wenhu Chen VLM MLLM 33 100 0 02 May 2024
Improved Baselines for Data-efficient Perceptual Augmentation of LLMs Théophane Vallaeys Mustafa Shukor Matthieu Cord Jakob Verbeek 54 12 0 20 Mar 2024
Chart-based Reasoning: Transferring Capabilities from LLMs to VLMs Victor Carbune Hassan Mansoor Fangyu Liu Rahul Aralikatte Gilles Baechler Jindong Chen Abhanshu Sharma ReLM LRM 133 12 0 19 Mar 2024
Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset Hugo Laurençon Léo Tronchon Victor Sanh VLM 52 36 0 14 Mar 2024
Gemma: Open Models Based on Gemini Research and Technology Gemma Team Gemma Team Thomas Mesnard Cassidy Hardin Robert Dadashi Surya Bhupatiraju ... Armand Joulin Noah Fiedel Evan Senter Alek Andreev Kathleen Kenealy VLM LLMAG 131 431 0 13 Mar 2024
MoAI: Mixture of All Intelligence for Large Language and Vision Models Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro MLLM VLM 45 20 0 12 Mar 2024
Efficient Multimodal Learning from Data-centric Perspective Muyang He Yexin Liu Boya Wu Jianhao Yuan Yueze Wang Tiejun Huang Bo-Lu Zhao MLLM 35 83 0 18 Feb 2024
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models Siddharth Karamcheti Suraj Nair Ashwin Balakrishna Percy Liang Thomas Kollar Dorsa Sadigh MLLM VLM 57 98 0 12 Feb 2024
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models Chris Liu Renrui Zhang Longtian Qiu Siyuan Huang Weifeng Lin ... Hao Shao Pan Lu Hongsheng Li Yu Qiao Peng Gao MLLM 130 109 0 08 Feb 2024
Enhancing Multimodal Large Language Models with Vision Detection Models: An Empirical Study Qirui Jiao Daoyuan Chen Yilun Huang Yaliang Li Ying Shen 28 15 0 31 Jan 2024
CogAgent: A Visual Language Model for GUI Agents Wenyi Hong Weihan Wang Qingsong Lv Jiazheng Xu Wenmeng Yu ... Juanzi Li Bin Xu Yuxiao Dong Ming Ding Jie Tang MLLM 142 321 0 14 Dec 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 270 4,244 0 30 Jan 2023
Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding Kenton Lee Mandar Joshi Iulia Turc Hexiang Hu Fangyu Liu Julian Martin Eisenschlos Urvashi Khandelwal Peter Shaw Ming-Wei Chang Kristina Toutanova CLIP VLM 163 263 0 07 Oct 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 211 1,106 0 20 Sep 2022
PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts Stephen H. Bach Victor Sanh Zheng-Xin Yong Albert Webson Colin Raffel ... Khalid Almubarak Xiangru Tang Dragomir R. Radev Mike Tian-Jian Jiang Alexander M. Rush VLM 225 338 0 02 Feb 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,137 0 28 Jan 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 213 1,657 0 15 Oct 2021
Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning Bryan Wang Gang Li Xin Zhou Zhourong Chen Tovi Grossman Yang Li 167 152 0 07 Aug 2021
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning Krishna Srinivasan K. Raman Jiecao Chen Michael Bendersky Marc Najork VLM 208 310 0 02 Mar 2021
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 278 1,082 0 17 Feb 2021