V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs

21 December 2023

Papers citing "V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs"

47 / 97 papers shown

Title
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark Wenhao Chai Enxin Song Y. Du Chenlin Meng Vashisht Madhavan Omer Bar-Tal Jeng-Neng Hwang Saining Xie Christopher D. Manning 3DV 77 25 0 04 Oct 2024
Effectively Enhancing Vision Language Large Models by Prompt Augmentation and Caption Utilization Minyi Zhao Jie Wang Z. Li Jiyuan Zhang Zhenbang Sun Shuigeng Zhou MLLM VLM 20 0 0 22 Sep 2024
A Survey on Multimodal Benchmarks: In the Era of Large AI Models Lin Li Guikun Chen Hanrong Shi Jun Xiao Long Chen 34 9 0 21 Sep 2024
Guided Profile Generation Improves Personalization with LLMs Jiarui Zhang 26 4 0 19 Sep 2024
A Survey on Evaluation of Multimodal Large Language Models Jiaxing Huang Jingyi Zhang LM&MA ELM LRM 46 20 0 28 Aug 2024
Visual Agents as Fast and Slow Thinkers Guangyan Sun Mingyu Jin Zhenting Wang Cheng-Long Wang Siqi Ma Qifan Wang Ying Nian Wu Ying Nian Wu Dongfang Liu Dongfang Liu LLMAG LRM 77 13 0 16 Aug 2024
FlexAttention for Efficient High-Resolution Vision-Language Models Junyan Li Delin Chen Tianle Cai Peihao Chen Yining Hong Zhenfang Chen Yikang Shen Chuang Gan VLM 67 4 0 29 Jul 2024
Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding Renshan Zhang Yibo Lyu Rui Shao Gongwei Chen Weili Guan Liqiang Nie 33 9 0 19 Jul 2024
Scaling 3D Reasoning with LMMs to Large Robot Mission Environments Using Datagraphs W. J. Meijer A. C. Kemmeren E.H.J. Riemens J. E. Fransman M. V. Bekkum G. J. Burghouts J. D. V. Mil 31 0 0 15 Jul 2024
HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Models Runhui Huang Xinpeng Ding Chunwei Wang J. N. Han Yulong Liu Hengshuang Zhao Hang Xu Lu Hou Wei Zhang Xiaodan Liang VLM 23 8 0 11 Jul 2024
Smart Vision-Language Reasoners Denisa Roberts Lucas Roberts VLM ReLM LRM 38 4 0 05 Jul 2024
Eliminating Position Bias of Language Models: A Mechanistic Approach Ziqi Wang Hanlin Zhang Xiner Li Kuan-Hao Huang Chi Han Shuiwang Ji Sham Kakade Hao Peng Heng Ji 49 12 0 01 Jul 2024
DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model Shezheng Song Shasha Li Jie Yu Shan Zhao Xiaopeng Li Jun Ma Xiaodong Liu Zhuo Li Xiaoguang Mao 21 1 0 27 Jun 2024
MM-SpuBench: Towards Better Understanding of Spurious Biases in Multimodal LLMs Wenqian Ye Guangtao Zheng Yunsheng Ma Xu Cao Bolin Lai James M. Rehg Aidong Zhang 33 10 0 24 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 37 278 0 24 Jun 2024
VLM Agents Generate Their Own Memories: Distilling Experience into Embodied Programs of Thought Gabriel H. Sarch Lawrence Jang Michael J. Tarr William W. Cohen Kenneth Marino Katerina Fragkiadaki LLMAG 36 0 0 20 Jun 2024
GUICourse: From General Vision Language Models to Versatile GUI Agents Wentong Chen Junbo Cui Jinyi Hu Yujia Qin Junjie Fang ... Yupeng Huo Yuan Yao Yankai Lin Zhiyuan Liu Maosong Sun LLMAG 31 30 0 17 Jun 2024
What is the Visual Cognition Gap between Humans and Multimodal LLMs? Xu Cao Bolin Lai Wenqian Ye Yunsheng Ma Joerg Heintz Jintai Chen Jianguo Cao James M. Rehg 45 8 0 14 Jun 2024
Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models Yushi Hu Weijia Shi Xingyu Fu Dan Roth Mari Ostendorf Luke Zettlemoyer Noah A. Smith Ranjay Krishna LRM 39 37 0 13 Jun 2024
Instruction-Guided Visual Masking Jinliang Zheng Jianxiong Li Si Cheng Yinan Zheng Jiaming Li Jihao Liu Yu Liu Jingjing Liu Xianyuan Zhan 45 5 0 30 May 2024
The Evolution of Multimodal Model Architectures S. Wadekar Abhishek Chaurasia Aman Chadha Eugenio Culurciello 41 14 0 28 May 2024
VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models Zejun Li Ruipu Luo Jiwen Zhang Minghui Qiu Zhongyu Wei Zhongyu Wei LRM MLLM 60 7 0 27 May 2024
Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model Kuan-Chih Huang Xiangtai Li Lu Qi Shuicheng Yan Ming-Hsuan Yang LRM 68 9 0 27 May 2024
LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate A. Fuller Daniel G. Kyrollos Yousef Yassin James R. Green 34 2 0 22 May 2024
Exploring the Capabilities of Large Multimodal Models on Dense Text Shuo Zhang Biao Yang Zhang Li Zhiyin Ma Yuliang Liu Xiang Bai VLM 29 7 0 09 May 2024
MARVEL: Multidimensional Abstraction and Reasoning through Visual Evaluation and Learning Yifan Jiang Jiarui Zhang Kexuan Sun Zhivar Sourati Kian Ahrabian Kaixin Ma Filip Ilievski Jay Pujara LRM 32 12 0 21 Apr 2024
TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding Bozhi Luan Hao Feng Hong Chen Yonghui Wang Wen-gang Zhou Houqiang Li MLLM 29 10 0 15 Apr 2024
Vision-Language Model-based Physical Reasoning for Robot Liquid Perception Wenqiang Lai Yuan Gao T. Lam LRM LM&Ro 25 5 0 10 Apr 2024
CoReS: Orchestrating the Dance of Reasoning and Segmentation Xiaoyi Bao Siyang Sun Shuailei Ma Kecheng Zheng Yuxin Guo Guosheng Zhao Yun Zheng Xingang Wang LRM 28 7 0 08 Apr 2024
Plug-and-Play Grounding of Reasoning in Multimodal Large Language Models Jiaxing Chen Yuxuan Liu Dehu Li Xiang An Weimo Deng Ziyong Feng Yongle Zhao Yin Xie LRM 38 14 0 28 Mar 2024
Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning Hao Shao Shengju Qian Han Xiao Guanglu Song Zhuofan Zong Letian Wang Yu Liu Hongsheng Li VGen LRM MLLM 58 37 0 25 Mar 2024
Pensieve: Retrospect-then-Compare Mitigates Visual Hallucination Dingchen Yang Bowen Cao Guang Chen Changjun Jiang 46 7 0 21 Mar 2024
When Do We Not Need Larger Vision Models? Baifeng Shi Ziyang Wu Maolin Mao Xin Wang Trevor Darrell VLM LRM 46 40 0 19 Mar 2024
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation Yunhao Gou Kai Chen Zhili Liu Lanqing Hong Hang Xu Zhenguo Li Dit-Yan Yeung James T. Kwok Yu Zhang MLLM 35 37 0 14 Mar 2024
BBA: Bi-Modal Behavioral Alignment for Reasoning with Large Vision-Language Models Xueliang Zhao Xinting Huang Tingchen Fu Qintong Li Shansan Gong Lemao Liu Wei Bi Lingpeng Kong LRM 33 1 0 21 Feb 2024
Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models Xuanyu Lei Zonghan Yang Xinrui Chen Peng Li Yang Liu MLLM LRM 32 30 0 19 Feb 2024
Exploring Perceptual Limitation of Multimodal Large Language Models Jiarui Zhang Jinyi Hu Mahyar Khayatkhoei Filip Ilievski Maosong Sun LRM 24 10 0 12 Feb 2024
V-IRL: Grounding Virtual Intelligence in Real Life Jihan Yang Runyu Ding Ellis L Brown Xiaojuan Qi Saining Xie LM&Ro 48 19 0 05 Feb 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 37 175 0 24 Jan 2024
REBUS: A Robust Evaluation Benchmark of Understanding Symbols Andrew Gritsevskiy Arjun Panickssery Aaron Kirtland Derik Kauffman Hans Gundlach Irina Gritsevskaya Joe Cavanagh Jonathan Chiang Lydia La Roux Michelle Hung ReLM 8 0 0 11 Jan 2024
GSVA: Generalized Segmentation via Multimodal Large Language Models Zhuofan Xia Dongchen Han Yizeng Han Xuran Pan Shiji Song Gao Huang VLM 23 54 0 15 Dec 2023
Towards Perceiving Small Visual Details in Zero-shot Visual Question Answering with Multimodal LLMs Jiarui Zhang Mahyar Khayatkhoei P. Chhikara Filip Ilievski 29 2 0 24 Oct 2023
Idea2Img: Iterative Self-Refinement with GPT-4V(ision) for Automatic Image Design and Generation Zhengyuan Yang Jianfeng Wang Linjie Li Kevin Qinghong Lin Chung-Ching Lin Zicheng Liu Lijuan Wang LRM MLLM DiffM 13 22 0 12 Oct 2023
ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models KAI-QING Zhou Kwonjoon Lee Teruhisa Misu Xin Eric Wang LRM 24 3 0 09 Oct 2023
A Survey on Multimodal Large Language Models Shukang Yin Chaoyou Fu Sirui Zhao Ke Li Xing Sun Tong Bill Xu Enhong Chen MLLM LRM 40 552 0 23 Jun 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 265 4,229 0 30 Jan 2023
Chain-of-Thought Prompting Elicits Reasoning in Large Language Models Jason W. Wei Xuezhi Wang Dale Schuurmans Maarten Bosma Brian Ichter F. Xia Ed H. Chi Quoc Le Denny Zhou LM&Ro LRM AI4CE ReLM 317 8,448 0 28 Jan 2022