Title
MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains Guoli Yin Haoping Bai Shuang Ma Feng Nan Yanchao Sun ... Xiaoming Wang Jiulong Shan Meng Cao Ruoming Pang Zirui Wang LLMAG ELM 79 7 0 18 Jul 2024
LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models Kaichen Zhang Bo Li Peiyuan Zhang Fanyi Pu Joshua Adrian Cahyono ... Shuai Liu Yuanhan Zhang Jingkang Yang Chunyuan Li Ziwei Liu 177 102 0 17 Jul 2024
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models Pengxiang Li Zhi Gao Bofei Zhang Tao Yuan Yuwei Wu Mehrtash Harandi Yunde Jia Song-Chun Zhu Qing Li VLM MLLM 102 6 0 16 Jul 2024
VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Models Haodong Duan Junming Yang Junming Yang Xinyu Fang Lin Chen ... Yuhang Zang Pan Zhang Jiaqi Wang Dahua Lin Kai Chen LM&MA VLM 246 180 0 16 Jul 2024
NTSEBENCH: Cognitive Reasoning Benchmark for Vision Language Models Pranshu Pandya Agney S Talwarr Vatsal Gupta Tushar Kataria Dan Roth Vivek Gupta LRM 153 2 0 15 Jul 2024
SoupLM: Model Integration in Large Language and Multi-Modal Models Yue Bai Zichen Zhang Jiasen Lu Yun Fu MoMe 62 1 0 11 Jul 2024
LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models Feng Li Renrui Zhang Hao Zhang Yuanhan Zhang Bo Li Wei Li Zejun Ma Chunyuan Li MLLM VLM 138 233 0 10 Jul 2024
Decompose and Compare Consistency: Measuring VLMs' Answer Reliability via Task-Decomposition Consistency Comparison Qian Yang Weixiang Yan Aishwarya Agrawal CoGe 79 4 0 10 Jul 2024
Multimodal Self-Instruct: Synthetic Abstract Image and Visual Reasoning Instruction Using Language Model Wenqi Zhang Zhenglin Cheng Yuanyu He Mengna Wang Yongliang Shen ... Guiyang Hou Mingqian He Yanna Ma Weiming Lu Yueting Zhuang SyDa 184 13 0 09 Jul 2024
A Single Transformer for Scalable Vision-Language Modeling Yangyi Chen Xingyao Wang Hao Peng Heng Ji LRM 107 17 0 08 Jul 2024
OmChat: A Recipe to Train Multimodal Language Models with Strong Long Context and Video Understanding Tiancheng Zhao Qianqian Zhang Kyusong Lee Peng Liu Lu Zhang Chunxin Fang Jiajia Liao Kelei Jiang Yibo Ma Ruochen Xu MLLM VLM 97 5 0 06 Jul 2024
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? Zhaorun Chen Yichao Du Zichen Wen Yiyang Zhou Chenhang Cui ... Jiawei Zhou Zhuokai Zhao Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM MLLM 117 35 0 05 Jul 2024
On scalable oversight with weak LLMs judging strong LLMs Zachary Kenton Noah Y. Siegel János Kramár Jonah Brown-Cohen Samuel Albanie ... Rishabh Agarwal David Lindner Yunhao Tang Noah D. Goodman Rohin Shah ELM 108 36 0 05 Jul 2024
Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding Xincan Feng Hidetaka Kamigaito Katsuhiko Hayashi Taro Watanabe 61 1 0 05 Jul 2024
Smart Vision-Language Reasoners Denisa Roberts Lucas Roberts VLM ReLM LRM 77 4 0 05 Jul 2024
A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations Md Tahmid Rahman Laskar Sawsan Alqahtani M Saiful Bari Mizanur Rahman Mohammad Abdullah Matin Khan ... Chee Wei Tan Md. Rizwan Parvez Enamul Hoque Shafiq Joty Jimmy Huang ELM ALM 105 41 0 04 Jul 2024
M5 -- A Diverse Benchmark to Assess the Performance of Large Multimodal Models Across Multilingual and Multicultural Vision-Language Tasks Florian Schneider Sunayana Sitaram VLM 81 12 0 04 Jul 2024
InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output Pan Zhang Xiaoyi Dong Yuhang Zang Yuhang Cao Rui Qian ... Kai Chen Jifeng Dai Yu Qiao Dahua Lin Jiaqi Wang 144 117 0 03 Jul 2024
VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values Zhe Hu Yixiao Ren Jing Li Yu Yin VLM 88 7 0 03 Jul 2024
TokenPacker: Efficient Visual Projector for Multimodal LLM Wentong Li Yuqian Yuan Jian Liu Dongqi Tang Song Wang Jie Qin Jianke Zhu Lei Zhang MLLM 148 67 0 02 Jul 2024
Synthetic Multimodal Question Generation Ian Wu Sravan Jayanthi Vijay Viswanathan Simon Rosenberg Sina Pakazad Tongshuang Wu Graham Neubig 91 5 0 02 Jul 2024
Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awareness Khyathi Chandu Linjie Li Anas Awadalla Ximing Lu Jae Sung Park Jack Hessel Lijuan Wang Yejin Choi 103 3 0 02 Jul 2024
VSP: Assessing the dual challenges of perception and reasoning in spatial planning tasks for VLMs Qiucheng Wu Handong Zhao Michael Stephen Saxon T. Bui William Yang Wang Yang Zhang Shiyu Chang CoGe 93 7 0 02 Jul 2024
We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning? Runqi Qiao Qiuna Tan Guanting Dong Minhui Wu Chong Sun ... Yida Xu Muxi Diao Zhimin Bao Chen Li Honggang Zhang VLM LRM 115 56 0 01 Jul 2024
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs Yusu Qian Hanrong Ye J. Fauconnier Peter Grasch Yinfei Yang Zhe Gan 249 18 0 01 Jul 2024
MMEvalPro: Calibrating Multimodal Benchmarks Towards Trustworthy and Efficient Evaluation Jinsheng Huang Liang Chen Taian Guo Fu Zeng Yusheng Zhao ... Wei Ju Luchen Liu Tianyu Liu Baobao Chang Ming Zhang 179 7 0 29 Jun 2024
Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs Sukmin Yun Haokun Lin Rusiru Thushara Mohammad Qazim Bhat Yongxin Wang ... Timothy Baldwin Zhengzhong Liu Eric P. Xing Xiaodan Liang Zhiqiang Shen 101 14 0 28 Jun 2024
LLaVolta: Efficient Multi-modal Models via Stage-wise Visual Context Compression Jieneng Chen Luoxin Ye Ju He Zhao-Yang Wang Daniel Khashabi Alan Yuille VLM 75 7 0 28 Jun 2024
STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Guohao Sun Can Qin Huazhu Fu Linwei Wang Zhiqiang Tao LM&MA 73 3 0 28 Jun 2024
From the Least to the Most: Building a Plug-and-Play Visual Reasoner via Data Synthesis Chuanqi Cheng Jian Guan Wei Wu Rui Yan LRM 95 13 0 28 Jun 2024
MMRo: Are Multimodal LLMs Eligible as the Brain for In-Home Robotics? Jinming Li Yichen Zhu Zhiyuan Xu Jindong Gu Minjie Zhu Xin Liu Ning Liu Yaxin Peng Feifei Feng Jian Tang LRM LM&Ro 103 8 0 28 Jun 2024
CMMaTH: A Chinese Multi-modal Math Skill Evaluation Benchmark for Foundation Models Zhong-Zhi Li Ming-Liang Zhang Fei Yin Zhi-Long Ji Jin-Feng Bai Zhen-Ru Pan Fan-Hu Zeng Jian Xu Jia-Xin Zhang Cheng-Lin Liu ELM 98 14 0 28 Jun 2024
HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scale Junying Chen Ruyi Ouyang Anningzhe Gao Shunian Chen Guiming Hardy Chen ... Zhenyang Cai Ke Ji Guangjun Yu Xiang Wan Benyou Wang MedIm LM&MA 82 50 0 27 Jun 2024
FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts Shubhankar Singh Purvi Chaurasia Yerram Varun Pranshu Pandya Vatsal Gupta Vivek Gupta Dan Roth 67 7 0 27 Jun 2024
ColPali: Efficient Document Retrieval with Vision Language Models Manuel Faysse Hugues Sibille Tony Wu Bilel Omrani Gautier Viaud C´eline Hudelot Pierre Colombo VLM 381 29 0 27 Jun 2024
S3: A Simple Strong Sample-effective Multimodal Dialog System Elisei Rykov Egor Malkershin Alexander Panchenko 87 0 0 26 Jun 2024
Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models Wenhao Shi Zhiqiang Hu Yi Bin Junhua Liu Yang Yang See-Kiong Ng Lidong Bing Roy Ka-Wei Lee SyDa MLLM LRM 115 62 0 25 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 166 377 0 24 Jun 2024
Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contexts Aditya Sharma Michael Saxon William Yang Wang VLM 70 2 0 24 Jun 2024
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models Mingrui Wu Jiayi Ji Oucheng Huang Jiale Li Yuhang Wu Xiaoshuai Sun Rongrong Ji 160 9 0 24 Jun 2024
AudioBench: A Universal Benchmark for Audio Large Language Models Bin Wang Xunlong Zou Geyu Lin Siyang Song Zhuohan Liu Wenyu Zhang Zhengyuan Liu AiTi Aw Nancy F. Chen AuLLM ELM LM&MA 169 35 0 23 Jun 2024
Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads A. Cherian Kuan-Chuan Peng Suhas Lohit Joanna Matthiesen Kevin A. Smith J. Tenenbaum ELM LRM 77 8 0 22 Jun 2024
Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs Yuxuan Qiao Haodong Duan Xinyu Fang Junming Yang Lin Chen Songyang Zhang Jiaqi Wang Dahua Lin Kai Chen LRM 107 23 0 20 Jun 2024
African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification Gregor Geigle Radu Timofte Goran Glavaš 83 13 0 20 Jun 2024
PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal Documents Junjie Wang Yin Zhang Yatai Ji Yuxiang Zhang Chunyang Jiang ... Bei Chen Qunshu Lin Minghao Liu Ge Zhang Wenhu Chen 97 3 0 20 Jun 2024
EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithms Siyu Yuan Kaitao Song Jiangjie Chen Xu Tan Dongsheng Li Deqing Yang LLMAG 178 28 0 20 Jun 2024
SpatialBot: Precise Spatial Understanding with Vision Language Models Wenxiao Cai Yaroslav Ponomarenko Jianhao Yuan Xiaoqi Li Wankou Yang Hao Dong Bo Zhao VLM 126 46 0 19 Jun 2024
Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning Bingchen Zhao Yongshuo Zong Letian Zhang Timothy Hospedales VLM 118 19 0 18 Jun 2024
OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AI Zhen Huang Zengzhi Wang Shijie Xia Xuefeng Li Haoyang Zou ... Yuxiang Zheng Shaoting Zhang Dahua Lin Yu Qiao Pengfei Liu ELM LRM 138 43 0 18 Jun 2024
Look Further Ahead: Testing the Limits of GPT-4 in Path Planning Mohamed Aghzal Erion Plaku Ziyu Yao ELM 102 7 0 17 Jun 2024