Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

11 January 2024

Shengbang Tong

Papers citing "Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs"

50 / 241 papers shown

Title
TIME: Temporal-sensitive Multi-dimensional Instruction Tuning and Benchmarking for Video-LLMs Yunxiao Wang Meng Liu Rui Shao Haoyu Zhang Bin Wen Fan Yang Tingting Gao Di Zhang Liqiang Nie 62 1 0 13 Mar 2025
Revisiting semi-supervised learning in the era of foundation models Ping Zhang Zheda Mai Quang-Huy Nguyen Wei-Lun Chao 52 0 0 12 Mar 2025
HaploVL: A Single-Transformer Baseline for Multi-Modal Understanding Rui Yang Lin Song Yicheng Xiao Runhui Huang Yixiao Ge Ying Shan Hengshuang Zhao MLLM 62 0 0 12 Mar 2025
Seeing What's Not There: Spurious Correlation in Multimodal LLMs Parsa Hosseini Sumit Nawathe Mazda Moayeri S. Balasubramanian S. Feizi LRM 41 1 0 11 Mar 2025
SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories Muzhi Zhu Yuzhuo Tian Hao Chen Chunluan Zhou Qingpei Guo Y. Liu M. Yang Chunhua Shen MLLM VLM 72 0 0 11 Mar 2025
Is CLIP ideal? No. Can we fix it? Yes! Raphi Kang Yue Song Georgia Gkioxari Pietro Perona VLM 58 0 0 10 Mar 2025
LLaVA-RadZ: Can Multimodal Large Language Models Effectively Tackle Zero-shot Radiology Recognition? Bangyan Li Wenxuan Huang Yunhang Shen Y. Wang Shaohui Lin ... Ling You Yinqi Zhang Ke Li Xing Sun Y. Sun 53 1 0 10 Mar 2025
DiffCLIP: Differential Attention Meets CLIP Hasan Hammoud Bernard Ghanem VLM 44 0 0 09 Mar 2025
What's in a Latent? Leveraging Diffusion Latent Space for Domain Generalization Xavier Thomas Deepti Ghadiyaram DiffM 92 0 0 09 Mar 2025
Is Your Video Language Model a Reliable Judge? M. Liu Wensheng Zhang 59 1 0 07 Mar 2025
Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting Dominic Maggio Luca Carlone 136 0 0 07 Mar 2025
See What You Are Told: Visual Attention Sink in Large Multimodal Models Seil Kang Jinyeong Kim Junhyeok Kim Seong Jae Hwang VLM 107 5 0 05 Mar 2025
Bridging VLM and KMP: Enabling Fine-grained robotic manipulation via Semantic Keypoints Representation Junjie Zhu Huayu Liu Jin Wang Bangrong Wen Kaixiang Huang Xiaofei Li Haiyun Zhan Guodong Lu 65 0 0 04 Mar 2025
Enhancing Vision-Language Compositional Understanding with Multimodal Synthetic Data Haoxin Li Boyang Li CoGe 73 0 0 03 Mar 2025
Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areas Shiqi Chen Tongyao Zhu Ruochen Zhou Jinghan Zhang Siyang Gao Juan Carlos Niebles Mor Geva Junxian He Jiajun Wu Manling Li LRM 60 0 0 03 Mar 2025
Evaluating and Predicting Distorted Human Body Parts for Generated Images Lu Ma Kaibo Cao Hao Liang Jiaxin Lin Z. Li Yuhong Liu Jihong Zhang Wentao Zhang Bin Cui MedIm 41 0 0 02 Mar 2025
Learning to Animate Images from A Few Videos to Portray Delicate Human Actions Haoxin Li Yingchen Yu Qilong Wu Hanwang Zhang Boyang Li Song Bai 3DH VGen 141 0 0 01 Mar 2025
Analyzing CLIP's Performance Limitations in Multi-Object Scenarios: A Controlled High-Resolution Study Reza Abbasi Ali Nazari Aminreza Sefid Mohammadali Banayeeanzade M. Rohban M. Baghshah VLM 56 1 0 27 Feb 2025
All-in-one: Understanding and Generation in Multimodal Reasoning with the MAIA Benchmark Davide Testa Giovanni Bonetta Raffaella Bernardi Alessandro Bondielli Alessandro Lenci Alessio Miaschi Lucia Passaro Bernardo Magnini VGen LRM 50 0 0 24 Feb 2025
LOVA3: Learning to Visual Question Answering, Asking and Assessment Henry Hengyuan Zhao Pan Zhou Difei Gao Zechen Bai Mike Zheng Shou 79 8 0 21 Feb 2025
Forgotten Polygons: Multimodal Large Language Models are Shape-Blind William Rudman Michal Golovanesky Amir Bar Vedant Palit Yann LeCun Carsten Eickhoff Ritambhara Singh LRM 49 2 0 21 Feb 2025
InsightVision: A Comprehensive, Multi-Level Chinese-based Benchmark for Evaluating Implicit Visual Semantics in Large Vision Language Models Xiaofei Yin Y. Hong Ya Guo Yi Tu Weiqiang Wang Gongshen Liu Huijia Zhu VLM 63 0 0 19 Feb 2025
Predicate Hierarchies Improve Few-Shot State Classification Emily Jin Joy Hsu Jiajun Wu OffRL 77 0 0 18 Feb 2025
Why Vision Language Models Struggle with Visual Arithmetic? Towards Enhanced Chart and Geometry Understanding Kung-Hsiang Huang Can Qin Haoyi Qiu Philippe Laban Shafiq R. Joty Caiming Xiong C. Wu VLM 147 1 0 17 Feb 2025
Investigating Inference-time Scaling for Chain of Multi-modal Thought: A Preliminary Study Yujie Lin Ante Wang Moye Chen Jingyao Liu Hao Liu Jinsong Su Xinyan Xiao LRM 48 2 0 17 Feb 2025
Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability Zhiyu Zhu Zhibo Jin Jiayu Zhang Nan Yang Jiahao Huang Jianlong Zhou Fang Chen 39 0 0 16 Feb 2025
Sparse Autoencoders for Scientifically Rigorous Interpretation of Vision Models Samuel Stevens Wei-Lun Chao T. Berger-Wolf Yu-Chuan Su VLM 72 2 0 10 Feb 2025
PixFoundation: Are We Heading in the Right Direction with Pixel-level Vision Foundation Models? Mennatullah Siam VLM 81 1 0 06 Feb 2025
Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models Chia-Wen Kuo Sijie Zhu Fan Chen Xiaohui Shen Longyin Wen VLM 65 1 0 04 Feb 2025
Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models H. Malik Fahad Shamshad Muzammal Naseer Karthik Nandakumar F. Khan Salman Khan AAML MLLM VLM 68 0 0 03 Feb 2025
SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training Tianzhe Chu Yuexiang Zhai Jihan Yang Shengbang Tong Saining Xie Dale Schuurmans Quoc V. Le Sergey Levine Yi-An Ma OffRL 70 56 0 28 Jan 2025
ViDDAR: Vision Language Model-Based Task-Detrimental Content Detection for Augmented Reality Yanming Xiu T. Scargill M. Gorlatova 70 2 0 22 Jan 2025
LEO: Boosting Mixture of Vision Encoders for Multimodal Large Language Models Mozhgan Nasr Azadani James Riddell Sean Sedwards Krzysztof Czarnecki MLLM VLM 44 2 0 13 Jan 2025
FOLDER: Accelerating Multi-modal Large Language Models with Enhanced Performance Haicheng Wang Zhemeng Yu Gabriele Spadaro Chen Ju Victor Quétu Enzo Tartaglione Enzo Tartaglione VLM 103 3 0 05 Jan 2025
Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models Ido Cohen Daniela Gottesman Mor Geva Raja Giryes VLM 92 0 1 18 Dec 2024
A Review of Multimodal Explainable Artificial Intelligence: Past, Present and Future Shilin Sun Wenbin An Feng Tian Fang Nan Qidong Liu J. Liu N. Shah Ping Chen 91 2 0 18 Dec 2024
Combating Multimodal LLM Hallucination via Bottom-Up Holistic Reasoning Shengqiong Wu Hao Fei Liangming Pan William Yang Wang Shuicheng Yan Tat-Seng Chua LRM 64 1 0 15 Dec 2024
FLAIR: VLM with Fine-grained Language-informed Image Representations Rui Xiao Sanghwan Kim Mariana-Iuliana Georgescu Zeynep Akata Stephan Alaniz VLM CLIP 71 2 0 04 Dec 2024
AdvDreamer Unveils: Are Vision-Language Models Truly Ready for Real-World 3D Variations? Shouwei Ruan Hanqin Liu Yao Huang Xiaoqi Wang Caixin Kang Hang Su Yinpeng Dong Xingxing Wei VGen 93 0 0 04 Dec 2024
ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation? Leixin Zhang Steffen Eger Yinjie Cheng Weihe Zhai Jonas Belouadi Christoph Leiter Simone Paolo Ponzetto Fahimeh Moafian Zhixue Zhao MLLM 76 1 0 03 Dec 2024
Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs Qizhe Zhang Aosong Cheng Ming Lu Zhiyong Zhuo Minqi Wang Jiajun Cao Shaobo Guo Qi She Shanghang Zhang VLM 90 11 0 02 Dec 2024
COSMOS: Cross-Modality Self-Distillation for Vision Language Pre-training Sanghwan Kim Rui Xiao Mariana-Iuliana Georgescu Stephan Alaniz Zeynep Akata VLM 74 1 0 02 Dec 2024
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones? Z. Chen Tingzhu Chen Wenjun Zhang Guangtao Zhai 93 3 0 02 Dec 2024
Talking to DINO: Bridging Self-Supervised Vision Backbones with Language for Open-Vocabulary Segmentation Luca Barsellotti Lorenzo Bianchi Nicola Messina F. Carrara Marcella Cornia Lorenzo Baraldi Fabrizio Falchi Rita Cucchiara VLM 72 2 0 28 Nov 2024
Evaluating Vision-Language Models as Evaluators in Path Planning Mohamed Aghzal Xiang Yue E. Plaku Ziyu Yao LRM 74 1 0 27 Nov 2024
NEMO: Can Multimodal LLMs Identify Attribute-Modified Objects? Jiaxuan Li Junwen Mo MinhDuc Vo Akihiro Sugimoto Hideki Nakayama 81 0 0 26 Nov 2024
Efficient Multi-modal Large Language Models via Visual Token Grouping Minbin Huang Runhui Huang Han Shi Yimeng Chen Chuanyang Zheng Xiangguo Sun Xin Jiang Z. Li Hong Cheng VLM 90 3 0 26 Nov 2024
Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge Yaqi Zhao Yuanyang Yin Lin Li Mingan Lin Victor Shea-Jay Huang Siwei Chen Weipeng Chen Baoqun Yin Zenan Zhou Wentao Zhang 77 0 0 25 Nov 2024
Is 'Right' Right? Enhancing Object Orientation Understanding in Multimodal Large Language Models through Egocentric Instruction Tuning Ji Hyeok Jung Eun Tae Kim S. Kim Joo Ho Lee Bumsoo Kim Buru Chang VLM 178 0 0 24 Nov 2024
$$\textit{Revelio}$: Interpreting and leveraging semantic information in diffusion models$ $\textit{Revelio}$ : Interpreting and leveraging semantic information in diffusion models Dahye Kim Xavier Thomas Deepti Ghadiyaram 83 4 0 23 Nov 2024