v1v2 (latest)

Law of Vision Representation in MLLMs

29 August 2024

Papers citing "Law of Vision Representation in MLLMs"

50 / 68 papers shown

Title
Analyzing Fine-Grained Alignment and Enhancing Vision Understanding in Multimodal Language Models Jiachen Jiang Jinxin Zhou Bo Peng Xia Ning Zhihui Zhu 93 0 0 22 May 2025
MIEB: Massive Image Embedding Benchmark Chenghao Xiao Isaac Chung Imene Kerboua Jamie Stirling Xin Zhang Márton Kardos Roman Solomatin Noura Al Moubayed Kenneth Enevoldsen Niklas Muennighoff VLM 101 2 0 14 Apr 2025
LangBridge: Interpreting Image as a Combination of Language Embeddings Jiaqi Liao Yuwei Niu Fanqing Meng Hao Li Changyao Tian ... Dianqi Li X. Zhu Li Yuan Jifeng Dai Yu Cheng MLLM 133 1 0 25 Mar 2025
Expanding the Boundaries of Vision Prior Knowledge in Multi-modal Large Language Models Qiao Liang Yanjiang Liu Xianpei Han Yaojie Lu Hongyu Lin Jia Zheng Jia Zheng Le Sun Le Sun Yingfei Sun 74 0 0 23 Mar 2025
TULIP: Towards Unified Language-Image Pretraining Zineng Tang Long Lian Seun Eisape Xudong Wang Roei Herzig Adam Yala Alane Suhr Trevor Darrell David M. Chan VLM CLIP MLLM 183 7 0 19 Mar 2025
HiDe-LLaVA: Hierarchical Decoupling for Continual Instruction Tuning of Multimodal Large Language Model Haiyang Guo Fanhu Zeng Ziwei Xiang Fei Zhu Da-Han Wang Xu-Yao Zhang Cheng-Lin Liu 116 3 0 17 Mar 2025
SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features Michael Tschannen A. Gritsenko Xiao Wang Muhammad Ferjad Naeem Ibrahim Alabdulmohsin ... Basil Mustafa Olivier J. Hénaff Jeremiah Harmsen Andreas Steiner Xiaohua Zhai VLM 139 80 0 21 Feb 2025
Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey Yunkai Dang Kaichen Huang Jiahao Huo Yibo Yan Shijie Huang ... Kun Wang Yong Liu Jing Shao Hui Xiong Xuming Hu LRM 144 22 0 03 Dec 2024
Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion Zhuokun Chen Jinwu Hu Zeshuai Deng Yufeng Wang Bohan Zhuang Mingkui Tan 109 0 0 02 Dec 2024
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Haotian Zhang Mingfei Gao Zhe Gan Philipp Dufter Nina Wenzel ... Haoxuan You Zirui Wang Afshin Dehghan Peter Grasch Yinfei Yang VLM MLLM 105 41 1 30 Sep 2024
Advancing Object Detection in Transportation with Multimodal Large Language Models (MLLMs): A Comprehensive Review and Empirical Testing Huthaifa I. Ashqar Ahmed Jaber Taqwa I. Alhadidi Mohammed Elhenawy 117 9 0 26 Sep 2024
NVLM: Open Frontier-Class Multimodal LLMs Wenliang Dai Nayeon Lee Wei Ping Zhuoling Yang Zihan Liu Jon Barker Tuomas Rintamaki Mohammad Shoeybi Bryan Catanzaro Ming-Yu Liu MLLM VLM LRM 83 73 0 17 Sep 2024
LLaVA-OneVision: Easy Visual Task Transfer Bo Li Yuanhan Zhang Dong Guo Renrui Zhang Feng Li Hao Zhang Kaichen Zhang Yanwei Li Ziwei Liu Chunyuan Li MLLM SyDa VLM 121 867 0 06 Aug 2024
Diffusion Feedback Helps CLIP See Better Wenxuan Wang Quan-Sen Sun Fan Zhang Yepeng Tang Jing Liu Xinlong Wang VLM 89 17 0 29 Jul 2024
Unified Lexical Representation for Interpretable Visual-Language Alignment Yifan Li Yikai Wang Yanwei Fu Dongyu Ru Zheng Zhang Tong He VLM 42 4 0 25 Jul 2024
RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulaiton Fanfan Liu Feng Yan Liming Zheng Chengjian Feng Yiyang Huang Lin Ma LM&Ro 82 12 0 27 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 126 378 0 24 Jun 2024
X-VILA: Cross-Modality Alignment for Large Language Model Hanrong Ye De-An Huang Yao Lu Zhiding Yu Ming-Yu Liu ... Jan Kautz Song Han Dan Xu Pavlo Molchanov Hongxu Yin MLLM VLM 80 35 0 29 May 2024
Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models Sangmin Woo Donguk Kim Jaehyuk Jang Yubin Choi Changick Kim 88 15 0 28 May 2024
ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models Chunjiang Ge Sijie Cheng Xiangqi Jin Jiale Yuan Yuan Gao Jun Song Shiji Song Gao Huang Bo Zheng MLLM VLM 69 17 0 24 May 2024
DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception Run Luo Yunshui Li Longze Chen Wanwei He Ting-En Lin ... Zikai Song Xiaobo Xia Tongliang Liu Min Yang Binyuan Hui VLM DiffM 135 22 0 24 May 2024
Focus Anywhere for Fine-grained Multi-page Document Understanding Chenglong Liu Haoran Wei Jinyue Chen Lingyu Kong Zheng Ge Zining Zhu Liang Zhao Jian‐Yuan Sun Chunrui Han Xiangyu Zhang 63 25 0 23 May 2024
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites Zhe Chen Weiyun Wang Hao Tian Shenglong Ye Zhangwei Gao ... Tong Lu Dahua Lin Yu Qiao Jifeng Dai Wenhai Wang MLLM VLM 115 642 0 25 Apr 2024
Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models Haotian Zhang Haoxuan You Philipp Dufter Bowen Zhang Chen Chen ... Tsu-Jui Fu William Y. Wang Shih-Fu Chang Zhe Gan Yinfei Yang ObjD MLLM 132 51 0 11 Apr 2024
BRAVE: Broadening the visual encoding of vision-language models Ouguzhan Fatih Kar A. Tonioni Petra Poklukar Achin Kulshrestha Amir Zamir Federico Tombari MLLM VLM 74 32 0 10 Apr 2024
DeepSeek-VL: Towards Real-World Vision-Language Understanding Haoyu Lu Wen Liu Bo Zhang Bing-Li Wang Kai Dong ... Yaofeng Sun Chengqi Deng Hanwei Xu Zhenda Xie Chong Ruan VLM 102 373 0 08 Mar 2024
Scaling Rectified Flow Transformers for High-Resolution Image Synthesis Patrick Esser Sumith Kulal A. Blattmann Rahim Entezari Jonas Muller ... Zion English Kyle Lacey Alex Goodwin Yannik Marek Robin Rombach DiffM 291 1,388 0 05 Mar 2024
InfiMM-HD: A Leap Forward in High-Resolution Multimodal Understanding Haogeng Liu Quanzeng You Xiaotian Han Yiqi Wang Bohan Zhai Yongfei Liu Yunzhe Tao Huaibo Huang Ran He Hongxia Yang MLLM 63 10 0 03 Mar 2024
Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models Siddharth Karamcheti Suraj Nair Ashwin Balakrishna Percy Liang Thomas Kollar Dorsa Sadigh MLLM VLM 112 133 0 12 Feb 2024
Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs Shengbang Tong Zhuang Liu Yuexiang Zhai Yi-An Ma Yann LeCun Saining Xie VLM MLLM 105 347 0 11 Jan 2024
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 260 1,215 0 21 Dec 2023
VILA: On Pre-training for Visual Language Models Ji Lin Hongxu Yin Ming-Yu Liu Yao Lu Pavlo Molchanov Andrew Tao Huizi Mao Jan Kautz Mohammad Shoeybi Song Han MLLM VLM 93 429 0 12 Dec 2023
Vary: Scaling up the Vision Vocabulary for Large Vision-Language Models Haoran Wei Lingyu Kong Jinyue Chen Liang Zhao Zheng Ge Jinrong Yang Jian‐Yuan Sun Chunrui Han Xiangyu Zhang MLLM VLM 110 75 0 11 Dec 2023
Telling Left from Right: Identifying Geometry-Aware Semantic Correspondence Junyi Zhang Charles Herrmann Junhwa Hur Eric Chen Varun Jampani Deqing Sun Ming-Hsuan Yang 73 45 0 28 Nov 2023
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 261 959 0 27 Nov 2023
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models Ziyi Lin Chris Liu Renrui Zhang Peng Gao Longtian Qiu ... Siyuan Huang Yichi Zhang Xuming He Hongsheng Li Yu Qiao MLLM VLM 87 230 0 13 Nov 2023
3DiffTection: 3D Object Detection with Geometry-Aware Diffusion Features Chenfeng Xu Huan Ling Sanja Fidler Or Litany 65 15 0 07 Nov 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 400 12,076 0 18 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Jiaqi Wang Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 123 1,059 0 12 Jul 2023
SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis Dustin Podell Zion English Kyle Lacey A. Blattmann Tim Dockhorn Jonas Muller Joe Penna Robin Rombach 255 2,447 0 04 Jul 2023
LightGlue: Local Feature Matching at Light Speed Philipp Lindenberger Paul-Edouard Sarlin Marc Pollefeys 3DV VLM 98 443 0 23 Jun 2023
MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models Chaoyou Fu Peixian Chen Yunhang Shen Yulei Qin Mengdan Zhang ... Xiawu Zheng Ke Li Xing Sun Zhenyu Qiu Rongrong Ji ELM MLLM 117 859 0 23 Jun 2023
OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents Hugo Laurenccon Lucile Saulnier Léo Tronchon Stas Bekman Amanpreet Singh ... Siddharth Karamcheti Alexander M. Rush Douwe Kiela Matthieu Cord Victor Sanh 132 246 0 21 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 432 4,422 0 09 Jun 2023
Emergent Correspondence from Image Diffusion Luming Tang Menglin Jia Qianqian Wang Cheng Perng Phoo Bharath Hariharan 99 267 0 06 Jun 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 571 4,910 0 17 Apr 2023
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab Timothée Darcet Théo Moutakanni Huy Q. Vo Marc Szafraniec ... Hervé Jégou Julien Mairal Patrick Labatut Armand Joulin Piotr Bojanowski VLM CLIP SSL 384 3,506 0 14 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 251 1,200 0 27 Mar 2023
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models Jiarui Xu Sifei Liu Arash Vahdat Wonmin Byeon Xiaolong Wang Shalini De Mello VLM 278 336 0 08 Mar 2023
HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention Shijie Geng Jianbo Yuan Yu Tian Yuxiao Chen Yongfeng Zhang CLIP VLM 68 46 0 06 Mar 2023