LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models

17 July 2024

Joshua Adrian Cahyono

Papers citing "LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models"

39 / 39 papers shown

Title
Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion Jacob A. Hansen Wei Lin Junmo Kang M. Jehanzeb Mirza Hongyin Luo Rogerio Feris Alan Ritter James R. Glass Leonid Karlinsky VLM 198 0 0 23 May 2025
Shadow-FT: Tuning Instruct via Base Taiqiang Wu Runming Yang Jiayi Li Pengfei Hu Ngai Wong Yujiu Yang 199 0 0 19 May 2025
QuantX: A Framework for Hardware-Aware Quantization of Generative AI Workloads Khurram Mazher Saad Bin Nasir MQ 79 0 0 12 May 2025
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment Yogesh Kulkarni Pooyan Fazli 73 0 0 18 Apr 2025
Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models Teppei Suzuki Keisuke Ozawa VLM 129 0 0 14 Apr 2025
SemHiTok: A Unified Image Tokenizer via Semantic-Guided Hierarchical Codebook for Multimodal Understanding and Generation Zhenpeng Chen Chunwei Wang Xiuwei Chen Hongbin Xu Jiawei Han Xiandan Liang J. N. Han Hang Xu Xiaodan Liang VLM 110 1 0 09 Mar 2025
DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models Saeed Ranjbar Alvar Gursimran Singh Mohammad Akbari Yong Zhang VLM 140 1 0 04 Mar 2025
Granite Vision: a lightweight, open-source multimodal model for enterprise Intelligence Granite Vision Team Leonid Karlinsky Assaf Arbelle Abraham Daniels A. Nassar ... Sriram Raghavan Tanveer Syeda-Mahmood Peter W. J. Staar Tal Drory Rogerio Feris VLM AI4TS 162 2 0 14 Feb 2025
Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding Ziyang Chen Mingxiao Li Zhongfu Chen Nan Du Xiaolong Li Yuexian Zou 86 1 0 19 Jan 2025
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 159 2 0 01 Dec 2024
JMMMU: A Japanese Massive Multi-discipline Multimodal Understanding Benchmark for Culture-aware Evaluation Shota Onohara Atsuyuki Miyai Yuki Imajuku Kazuki Egashira Jeonghun Baek Xiang Yue Graham Neubig Kiyoharu Aizawa OSLM 153 6 0 22 Oct 2024
3DArticCyclists: Generating Synthetic Articulated 8D Pose-Controllable Cyclist Data for Computer Vision Applications Eduardo R. Corral-Soto Yang Liu Tongtong Cao Y. Ren Liu Bingbing 85 5 0 14 Oct 2024
Can We Predict Performance of Large Models across Vision-Language Tasks? Qinyu Zhao Ming Xu Kartik Gupta Akshay Asthana Liang Zheng Stephen Gould 91 0 0 14 Oct 2024
LLaVA-Critic: Learning to Evaluate Multimodal Models Tianyi Xiong Xinze Wang Dong Guo Qinghao Ye Haoqi Fan Quanquan Gu Heng Huang Chunyuan Li MLLM VLM LRM 89 45 0 03 Oct 2024
A-VL: Adaptive Attention for Large Vision-Language Models Junyang Zhang Mu Yuan Ruiguang Zhong Puhan Luo Huiyou Zhan Ningkang Zhang Chengchen Hu Xiangyang Li VLM 81 1 0 23 Sep 2024
Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models Aitor Ormazabal Che Zheng Cyprien de Masson dÁutume Dani Yogatama Deyu Fu ... Yazheng Yang Yi Tay Yuqi Wang Zhongkai Zhu Zhihui Xie LRM VLM ReLM 62 51 0 18 Apr 2024
InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HD Xiao-wen Dong Pan Zhang Yuhang Zang Yuhang Cao Bin Wang ... Xingcheng Zhang Jifeng Dai Yuxin Qiao Dahua Lin Jiaqi Wang VLM MLLM 74 122 0 09 Apr 2024
Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference Wei-Lin Chiang Lianmin Zheng Ying Sheng Anastasios Nikolas Angelopoulos Tianle Li ... Hao Zhang Banghua Zhu Michael I. Jordan Joseph E. Gonzalez Ion Stoica OSLM 127 567 0 07 Mar 2024
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 188 891 0 27 Nov 2023
Rethinking Benchmark and Contamination for Language Models with Rephrased Samples Shuo Yang Wei-Lin Chiang Lianmin Zheng Joseph E. Gonzalez Ion Stoica ALM 53 125 0 08 Nov 2023
Skywork: A More Open Bilingual Foundation Model Tianwen Wei Liang Zhao Lichang Zhang Bo Zhu Lijie Wang ... Yongyi Peng Xiaojuan Liang Shuicheng Yan Han Fang Yahui Zhou 65 98 0 30 Oct 2023
Ferret: Refer and Ground Anything Anywhere at Any Granularity Haoxuan You Haotian Zhang Zhe Gan Xianzhi Du Bowen Zhang Zirui Wang Liangliang Cao Shih-Fu Chang Yinfei Yang ObjD MLLM VLM 87 316 0 11 Oct 2023
MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts Pan Lu Hritik Bansal Tony Xia Jiacheng Liu Chun-yue Li Hannaneh Hajishirzi Hao Cheng Kai-Wei Chang Michel Galley Jianfeng Gao LRM MLLM 91 605 0 03 Oct 2023
InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition Pan Zhang Xiaoyi Wang Bin Wang Yuhang Cao Chao Xu ... Conghui He Xingcheng Zhang Yu Qiao Da Lin Jiaqi Wang MLLM 118 235 0 26 Sep 2023
Hierarchical multimodal transformers for Multi-Page DocVQA Rubèn Pérez Tito Dimosthenis Karatzas Ernest Valveny 52 59 0 07 Dec 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 250 1,235 0 20 Sep 2022
ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning Ahmed Masry Do Xuan Long J. Tan Shafiq Joty Enamul Hoque AIMat 92 649 0 19 Mar 2022
IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning Pan Lu Liang Qiu Jiaqi Chen Tony Xia Yizhou Zhao Wei Zhang Zhou Yu Xiaodan Liang Song-Chun Zhu AIMat 80 198 0 25 Oct 2021
InfographicVQA Minesh Mathew Viraj Bagal Rubèn Pérez Tito Dimosthenis Karatzas Ernest Valveny C. V. Jawahar 80 228 0 26 Apr 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 824 29,167 0 26 Feb 2021
WebSRC: A Dataset for Web-Based Structural Reading Comprehension Xingyu Chen Zihan Zhao Lu Chen Danyang Zhang Jiabao Ji Ao Luo Yuxuan Xiong Kai Yu RALM 55 95 0 23 Jan 2021
DocVQA: A Dataset for VQA on Document Images Minesh Mathew Dimosthenis Karatzas C. V. Jawahar 119 714 0 01 Jul 2020
TextCaps: a Dataset for Image Captioning with Reading Comprehension Oleksii Sidorov Ronghang Hu Marcus Rohrbach Amanpreet Singh 58 411 0 24 Mar 2020
Scene Text Visual Question Answering Ali Furkan Biten Rubèn Pérez Tito Andrés Mafla Lluís Gómez Marçal Rusiñol Ernest Valveny C. V. Jawahar Dimosthenis Karatzas 82 355 0 31 May 2019
Towards VQA Models That Can Read Amanpreet Singh Vivek Natarajan Meet Shah Yu Jiang Xinlei Chen Dhruv Batra Devi Parikh Marcus Rohrbach EgoV 69 1,215 0 18 Apr 2019
VizWiz Grand Challenge: Answering Visual Questions from Blind People Danna Gurari Qing Li Abigale Stangl Anhong Guo Chi Lin Kristen Grauman Jiebo Luo Jeffrey P. Bigham CoGe 82 847 0 22 Feb 2018
Understanding Black-box Predictions via Influence Functions Pang Wei Koh Percy Liang TDI 169 2,878 0 14 Mar 2017
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Yash Goyal Tejas Khot D. Summers-Stay Dhruv Batra Devi Parikh CoGe 322 3,224 0 02 Dec 2016
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 385 43,524 0 01 May 2014