Kosmos-2: Grounding Multimodal Large Language Models to the World

26 June 2023

Papers citing "Kosmos-2: Grounding Multimodal Large Language Models to the World"

50 / 142 papers shown

Title
Do Vision-Language Models Represent Space and How? Evaluating Spatial Frame of Reference Under Ambiguities Zheyuan Zhang Fengyuan Hu Jayjun Lee Freda Shi Parisa Kordjamshidi Joyce Chai Ziqiao Ma 59 11 0 22 Oct 2024
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment Chenhang Cui An Zhang Yiyang Zhou Zhaorun Chen Gelei Deng Huaxiu Yao Tat-Seng Chua 73 4 0 18 Oct 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Zou Tatsunori Hashimoto VLM 70 4 0 14 Oct 2024
Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training Gen Luo Xue Yang Wenhan Dou Zhaokai Wang Jifeng Dai Jifeng Dai Yu Qiao Xizhou Zhu VLM MLLM 65 25 0 10 Oct 2024
Compositional Entailment Learning for Hyperbolic Vision-Language Models Avik Pal Max van Spengler Guido Maria DÁmely di Melendugno Alessandro Flaborea Fabio Galasso Pascal Mettes CoGe 48 5 0 09 Oct 2024
Organizing Unstructured Image Collections using Natural Language Mingxuan Liu Zhun Zhong Jun Li Gianni Franchi Subhankar Roy Elisa Ricci VLM 39 3 0 07 Oct 2024
Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents Boyu Gou Ruohan Wang Boyuan Zheng Yanan Xie Cheng Chang Yiheng Shu Huan Sun Yu Su LM&Ro LLMAG 78 49 0 07 Oct 2024
Is What You Ask For What You Get? Investigating Concept Associations in Text-to-Image Models Salma Abdel Magid Weiwei Pan Simon Warchol Grace Guo Junsik Kim Mahia Rahman Hanspeter Pfister 95 0 0 06 Oct 2024
Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructions Lingwei Meng Shujie Hu Jiawen Kang Zhaoqing Li Yuejiao Wang Wenxuan Wu Xixin Wu Xunying Liu Helen Meng AuLLM 70 2 0 13 Sep 2024
Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding Yunze Man Shuhong Zheng Zhipeng Bao M. Hebert Liang-Yan Gui Yu-xiong Wang 75 15 0 05 Sep 2024
SAM4MLLM: Enhance Multi-Modal Large Language Model for Referring Expression Segmentation Yi-Chia Chen Wei-Hua Li Cheng Sun Yu-Chiang Frank Wang Chu-Song Chen VLM 39 11 0 01 Sep 2024
IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities Bin Wang Chunyu Xie Dawei Leng Yuhui Yin MLLM 54 1 0 23 Aug 2024
Visual Agents as Fast and Slow Thinkers Guangyan Sun Mingyu Jin Zhenting Wang Cheng-Long Wang Siqi Ma Qifan Wang Ying Nian Wu Ying Nian Wu Dongfang Liu Dongfang Liu LLMAG LRM 79 13 0 16 Aug 2024
LLMServingSim: A HW/SW Co-Simulation Infrastructure for LLM Inference Serving at Scale Jaehong Cho Minsu Kim Hyunmin Choi Guseul Heo Jongse Park 38 9 0 10 Aug 2024
GalleryGPT: Analyzing Paintings with Large Multimodal Models Yi Bin Wenhao Shi Yujuan Ding Zhiqiang Hu Zheng Wang Yang Yang See-Kiong Ng H. Shen MLLM 30 11 0 01 Aug 2024
ControlMLLM: Training-Free Visual Prompt Learning for Multimodal Large Language Models Ming-Kuan Wu Xinyue Cai Jiayi Ji Jiale Li Oucheng Huang Gen Luo Hao Fei Xiaoshuai Sun Rongrong Ji MLLM 51 7 0 31 Jul 2024
Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing Ekaterina Iakovleva Fabio Pizzati Philip H. S. Torr Stéphane Lathuiliere DiffM 31 0 0 29 Jul 2024
Learning Visual Grounding from Generative Vision and Language Model Shijie Wang Dahun Kim A. Taalimi Chen Sun Weicheng Kuo ObjD 36 5 0 18 Jul 2024
ViLLa: Video Reasoning Segmentation with Large Language Model Rongkun Zheng Lu Qi Xi Chen Yi Wang Kun Wang Yu Qiao Hengshuang Zhao VOS LRM 75 2 0 18 Jul 2024
A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding Jinghui Lu Haiyang Yu Yunhong Wang Yongjie Ye Jingqun Tang ... Qi Liu Hao Feng Hairu Wang Hao Liu Can Huang 50 18 0 02 Jul 2024
MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMs Yusu Qian Hanrong Ye J. Fauconnier Peter Grasch Yinfei Yang Zhe Gan 108 13 0 01 Jul 2024
$GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing$ GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing Yisong Xiao Aishan Liu QianJia Cheng Zhenfei Yin Siyuan Liang Jiapeng Li Jing Shao Xianglong Liu Dacheng Tao 48 4 0 30 Jun 2024
Urban Visual Appeal According to ChatGPT: Contrasting AI and Human Insights M. Malekzadeh Elias S Willberg Jussi Torkko T. Toivonen 43 1 0 29 Jun 2024
LLaRA: Supercharging Robot Learning Data for Vision-Language Policy Xiang Li Cristina Mata J. Park Kumara Kahatapitiya Yoo Sung Jang ... Kanchana Ranasinghe R. Burgert Mu Cai Yong Jae Lee Michael S. Ryoo LM&Ro 72 25 0 28 Jun 2024
Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Language Yicheng Chen Xiangtai Li Yining Li Yanhong Zeng Jianzong Wu Xiangyu Zhao Kai Chen VLM DiffM 56 3 0 28 Jun 2024
Reminding Multimodal Large Language Models of Object-aware Knowledge with Retrieved Tags Daiqing Qi Handong Zhao Zijun Wei Sheng Li 46 2 0 16 Jun 2024
VLind-Bench: Measuring Language Priors in Large Vision-Language Models Kang-il Lee Minbeom Kim Seunghyun Yoon Minsung Kim Dongryeol Lee Hyukhun Koh Kyomin Jung CoGe VLM 92 5 0 13 Jun 2024
Grounding Multimodal Large Language Models in Actions Andrew Szot Bogdan Mazoure Harsh Agrawal Devon Hjelm Z. Kira Alexander Toshev LM&Ro 35 10 0 12 Jun 2024
MLLMGuard: A Multi-dimensional Safety Evaluation Suite for Multimodal Large Language Models Tianle Gu Zeyang Zhou Kexin Huang Dandan Liang Yixu Wang ... Keqing Wang Yujiu Yang Yan Teng Yu Qiao Yingchun Wang ELM 50 13 0 11 Jun 2024
Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation Peize Sun Yi Jiang Shoufa Chen Shilong Zhang Bingyue Peng Ping Luo Zehuan Yuan VLM 66 227 0 10 Jun 2024
F-LMM: Grounding Frozen Large Multimodal Models Size Wu Sheng Jin Wenwei Zhang Lumin Xu Wentao Liu Wei Li Chen Change Loy MLLM 80 12 0 09 Jun 2024
3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination Jianing Yang Xuweiyi Chen Nikhil Madaan Madhavan Iyengar Shengyi Qian David Fouhey Joyce Chai 3DV 75 11 0 07 Jun 2024
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models Zejun Li Ruipu Luo Jiwen Zhang Minghui Qiu Zhongyu Wei Zhongyu Wei LRM MLLM 62 7 0 27 May 2024
Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models Yue Zhang Hehe Fan Yi Yang 53 3 0 24 May 2024
V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM Abdur Rahman Rajat Chawla Muskaan Kumar Arkajit Datta Adarsh Jha NS Mukunda Ishaan Bhola 48 2 0 24 May 2024
DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception Run Luo Yunshui Li Longze Chen Wanwei He Ting-En Lin ... Zikai Song Xiaobo Xia Tongliang Liu Min Yang Binyuan Hui VLM DiffM 75 15 0 24 May 2024
A Survey on Vision-Language-Action Models for Embodied AI Yueen Ma Zixing Song Yuzheng Zhuang Jianye Hao Irwin King LM&Ro 82 42 0 23 May 2024
ReasonPix2Pix: Instruction Reasoning Dataset for Advanced Image Editing Ying Jin Pengyang Ling Xiao-wen Dong Pan Zhang Jiaqi Wang Dahua Lin 34 2 0 18 May 2024
SpeechVerse: A Large-scale Generalizable Audio Language Model Nilaksh Das Saket Dingliwal S. Ronanki Rohit Paturi David Huang ... Monica Sunkara S. Srinivasan Kyu J. Han Katrin Kirchhoff Katrin Kirchhoff 41 37 0 14 May 2024
G-VOILA: Gaze-Facilitated Information Querying in Daily Scenarios Zeyu Wang Yuanchun Shi Yuntao wang Yuchen Yao Kun Yan Yuhan Wang Lei Ji Xuhai Xu Chun Yu 40 7 0 13 May 2024
THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models Prannay Kaul Zhizhong Li Hao Yang Yonatan Dukler Ashwin Swaminathan C. Taylor Stefano Soatto HILM 60 16 0 08 May 2024
List Items One by One: A New Data Source and Learning Paradigm for Multimodal LLMs An Yan Zhengyuan Yang Junda Wu Wanrong Zhu Jianwei Yang ... K. Lin Jianfeng Wang Julian McAuley Jianfeng Gao Lijuan Wang LRM 34 12 0 25 Apr 2024
UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation Siru Zhong Xixuan Hao Yibo Yan Ying Zhang Yangqiu Song Yuxuan Liang 42 8 0 22 Apr 2024
MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering Avinash Anand Janak Kapuriya Chhavi Kirtani Apoorv Singh Jay Saraf Naman Lal Jatin Kumar A. Shivam Astha Verma R. Shah OffRL 45 9 0 19 Apr 2024
HOI-Ref: Hand-Object Interaction Referral in Egocentric Vision Siddhant Bansal Michael Wray Dima Damen 41 3 0 15 Apr 2024
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs Keen You Haotian Zhang E. Schoop Floris Weers Amanda Swearngin Jeffrey Nichols Yinfei Yang Zhe Gan MLLM 47 82 0 08 Apr 2024
Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models Jesse Atuhurra Iqra Ali Tatsuya Hiraoka Hidetaka Kamigaito Tomoya Iwakura Taro Watanabe 44 1 0 29 Mar 2024
Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want Weifeng Lin Xinyu Wei Ruichuan An Peng Gao Bocheng Zou Yulin Luo Siyuan Huang Shanghang Zhang Hongsheng Li VLM 69 33 0 29 Mar 2024
RH20T-P: A Primitive-Level Robotic Dataset Towards Composable Generalization Agents Zeren Chen Zhelun Shi Xiaoya Lu Lehan He Sucheng Qian ... Zhen-fei Yin Jing Shao Jing Shao Cewu Lu Cewu Lu 38 5 0 28 Mar 2024
A Modular Approach for Multimodal Summarization of TV Shows Louis Mahon Mirella Lapata 26 9 0 06 Mar 2024