What matters when building vision-language models?

3 May 2024

Papers citing "What matters when building vision-language models?"

50 / 133 papers shown

Title
Vision language models have difficulty recognizing virtual objects Tyler Tran Sangeet Khemlani J. Gregory Trafton 14 0 0 15 May 2025
DriveSOTIF: Advancing Perception SOTIF Through Multimodal Large Language Models Shucheng Huang Freda Shi Chen Sun Jiaming Zhong Minghao Ning Yufeng Yang Yukun Lu Hong Wang A. Khajepour 31 0 0 11 May 2025
R^3-VQA: "Read the Room" by Video Social Reasoning Lixing Niu Jiapeng Li Xingping Yu Shu Wang Ruining Feng Bo Wu Ping Wei Y. Wang Lifeng Fan 51 0 0 07 May 2025
Perception Encoder: The best visual embeddings are not at the output of the network Daniel Bolya Po-Yao (Bernie) Huang Peize Sun Jang Hyun Cho Andrea Madotto ... Shiyu Dong Nikhila Ravi Daniel Li Piotr Dollár Christoph Feichtenhofer ObjD VOS 103 0 0 17 Apr 2025
Mimic In-Context Learning for Multimodal Tasks Yuchu Jiang Jiale Fu Chenduo Hao Xinting Hu Yingzhe Peng Xin Geng Xu Yang 27 0 0 11 Apr 2025
Data Metabolism: An Efficient Data Design Schema For Vision Language Model Jingyuan Zhang Hongzhi Zhang Zhou Haonan Chenxi Sun Xingguang Ji Jiakang Wang Fanheng Kong Y. Liu Qi Wang Fuzheng Zhang VLM 60 1 0 10 Apr 2025
Classifying the Unknown: In-Context Learning for Open-Vocabulary Text and Symbol Recognition Tom Simon William Mocaer Pierrick Tranouez Clément Chatelain Thierry Paquet MLLM VLM 51 0 0 09 Apr 2025
SmolVLM: Redefining small and efficient multimodal models Andres Marafioti Orr Zohar Miquel Farré Merve Noyan Elie Bakouch ... Hugo Larcher Mathieu Morlon Lewis Tunstall Leandro von Werra Thomas Wolf VLM 39 6 0 07 Apr 2025
M2IV: Towards Efficient and Fine-grained Multimodal In-Context Learning in Large Vision-Language Models Yanshu Li Hongyang He Yi Cao Qisen Cheng Xiang Fu Ruixiang Tang VLM 40 0 0 06 Apr 2025
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization Chaohu Liu Tianyi Gui Yu Liu Linli Xu VLM AAML 68 1 0 02 Apr 2025
Open-Qwen2VL: Compute-Efficient Pre-Training of Fully-Open Multimodal LLMs on Academic Resources Weizhi Wang Yu Tian L. Yang Heng Wang Xifeng Yan MLLM VLM 79 0 0 01 Apr 2025
On Large Multimodal Models as Open-World Image Classifiers Alessandro Conti Massimiliano Mancini Enrico Fini Yiming Wang Paolo Rota Elisa Ricci VLM Presented at ResearchTrend Connect \| VLM on 07 May 2025 84 0 0 27 Mar 2025
MLLM-Selector: Necessity and Diversity-driven High-Value Data Selection for Enhanced Visual Instruction Tuning Yiwei Ma Guohai Xu Xiaoshuai Sun Jiayi Ji Jie Lou Debing Zhang Rongrong Ji 92 0 0 26 Mar 2025
On the Perception Bottleneck of VLMs for Chart Understanding Junteng Liu Weihao Zeng Xiwen Zhang Yijun Wang Zifei Shan Junxian He 60 0 0 24 Mar 2025
PVChat: Personalized Video Chat with One-Shot Learning Yufei Shi Weilong Yan Gang Xu Yumeng Li Y. Li Z. Li Fei Richard Yu Ming Li Si Yong Yeo 43 0 0 21 Mar 2025
VIPER: Visual Perception and Explainable Reasoning for Sequential Decision-Making Mohamed Salim Aissi Clemence Grislain Mohamed Chetouani Olivier Sigaud Laure Soulier Nicolas Thome LRM 42 0 0 19 Mar 2025
Identifying and Mitigating Position Bias of Multi-image Vision-Language Models Xinyu Tian Shu Zou Zhaoyuan Yang Jing Zhang 63 0 0 18 Mar 2025
Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives Sara Sarto Marcella Cornia Rita Cucchiara 46 0 0 18 Mar 2025
HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Model Tao Wang Changxu Cheng Lingfeng Wang Senda Chen Wuyue Zhao VLM 72 0 0 17 Mar 2025
Efficient Motion-Aware Video MLLM Zijia Zhao Yuqi Huo Tongtian Yue Longteng Guo Haoyu Lu B. Wang Weipeng Chen J. Liu 60 0 0 17 Mar 2025
VideoMind: A Chain-of-LoRA Agent for Long Video Reasoning Y. Liu Kevin Qinghong Lin C. Chen Mike Zheng Shou LM&Ro LRM 84 0 0 17 Mar 2025
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs Erik Daxberger Nina Wenzel David Griffiths Haiming Gang Justin Lazarow ... Kai Kang Marcin Eichner Y. Yang Afshin Dehghan Peter Grasch 72 3 0 17 Mar 2025
DeepPerception: Advancing R1-like Cognitive Visual Perception in MLLMs for Knowledge-Intensive Visual Grounding Xinyu Ma Ziyang Ding Zhicong Luo C. L. P. Chen Zonghao Guo Derek F. Wong Xiaoyi Feng Maosong Sun VLM LRM 76 0 0 17 Mar 2025
Similarity-Aware Token Pruning: Your VLM but Faster Ahmadreza Jeddi Negin Baghbanzadeh Elham Dolatabadi Babak Taati 3DV VLM 56 1 0 14 Mar 2025
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers Weiming Ren Wentao Ma Huan Yang Cong Wei Ge Zhang Wenhu Chen Mamba 57 3 0 14 Mar 2025
TikZero: Zero-Shot Text-Guided Graphics Program Synthesis Jonas Belouadi Eddy Ilg M. Keuper Hideki Tanaka Masao Utiyama Raj Dabre Steffen Eger Simone Paolo Ponzetto 50 0 0 14 Mar 2025
Cockatiel: Ensembling Synthetic and Human Preferenced Training for Detailed Video Caption Luozheng Qin Zhiyu Tan Mengping Yang Xiaomeng Yang Hao Li 84 0 0 12 Mar 2025
Should VLMs be Pre-trained with Image Data? Sedrick Scott Keh Jean-Pierre Mercat S. Gadre Kushal Arora Igor Vasiljevic ... Shuran Song Russ Tedrake Thomas Kollar Ludwig Schmidt Achal Dave VLM 49 0 0 10 Mar 2025
A Novel Trustworthy Video Summarization Algorithm Through a Mixture of LoRA Experts Wenzhuo Du G. Wang Guancheng Chen Hang Zhao X. Li Jian Gao 155 0 0 08 Mar 2025
OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction Huang Huang Fangchen Liu Letian Fu Tingfan Wu Mustafa Mukadam Jitendra Malik Ken Goldberg Pieter Abbeel LM&Ro VLM 82 5 0 05 Mar 2025
A Token-level Text Image Foundation Model for Document Understanding Tongkun Guan Zining Wang Pei Fu Zhengtao Guo Wei-Ming Shen ... Chen Duan Hao Sun Qianyi Jiang Junfeng Luo Xiaokang Yang VLM 45 0 0 04 Mar 2025
T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generation Lijun Li Zhelun Shi Xuhao Hu Bowen Dong Yiran Qin Xihui Liu Lu Sheng Jing Shao 112 1 0 21 Feb 2025
Chitrarth: Bridging Vision and Language for a Billion People Shaharukh Khan Ayush Tarun Abhinav Ravi Ali Faraz Akshat Patidar Praveen Kumar Pokala Anagha Bhangare Raja Kolla Chandra Khatri Shubham Agarwal VLM 120 1 0 21 Feb 2025
Rethinking Homogeneity of Vision and Text Tokens in Large Vision-and-Language Models Chia-Wen Kuo Sijie Zhu Fan Chen Xiaohui Shen Longyin Wen VLM 65 1 0 04 Feb 2025
AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Understanding Ahmed Masry Juan A. Rodriguez Tianyu Zhang Suyuchen Wang Chao Wang ... I. Laradji David Vazquez Perouz Taslakian Spandana Gella Sai Rajeswar 51 0 0 03 Feb 2025
Baichuan-Omni-1.5 Technical Report Yadong Li J. Liu Tao Zhang Tao Zhang S. Chen ... Jianhua Xu Haoze Sun Mingan Lin Zenan Zhou Weipeng Chen AuLLM 72 10 0 28 Jan 2025
MMVU: Measuring Expert-Level Multi-Discipline Video Understanding Yilun Zhao Lujing Xie Haowei Zhang Guo Gan Yitao Long ... Xiangru Tang Zhenwen Liang Y. Liu Chen Zhao Arman Cohan 53 5 0 21 Jan 2025
CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification Cristiano Patrício Isabel Rio-Torto J. S. Cardoso Luís F. Teixeira João C. Neves VLM 215 0 0 21 Jan 2025
Analyzing Fine-tuning Representation Shift for Multimodal LLMs Steering alignment Pegah Khayatan Mustafa Shukor Jayneel Parekh Matthieu Cord LLMSV 41 1 0 06 Jan 2025
Automated Generation of Challenging Multiple-Choice Questions for Vision Language Model Evaluation Yuhui Zhang Yuchang Su Yiming Liu Xiaohan Wang James Burgess ... Josiah Aklilu Alejandro Lozano Anjiang Wei Ludwig Schmidt Serena Yeung-Levy 61 3 0 06 Jan 2025
2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining Wenqi Zhang Hang Zhang Xin Li Jiashuo Sun Yongliang Shen Weiming Lu Deli Zhao Yueting Zhuang Lidong Bing VLM 41 2 0 01 Jan 2025
Multimodal Preference Data Synthetic Alignment with Reward Model Robert Wijaya Ngoc-Bao Nguyen Ngai-man Cheung MLLM SyDa 62 2 0 23 Dec 2024
Progress-Aware Video Frame Captioning Zihui Xue Joungbin An Xitong Yang Kristen Grauman 100 1 0 03 Dec 2024
OBI-Bench: Can LMMs Aid in Study of Ancient Script on Oracle Bones? Z. Chen Tingzhu Chen Wenjun Zhang Guangtao Zhai 96 3 0 02 Dec 2024
Beyond Sight: Towards Cognitive Alignment in LVLM via Enriched Visual Knowledge Yaqi Zhao Yuanyang Yin Lin Li Mingan Lin Victor Shea-Jay Huang Siwei Chen Weipeng Chen Baoqun Yin Zenan Zhou Wentao Zhang 77 0 0 25 Nov 2024
SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization Hongrui Jia Chaoya Jiang Haiyang Xu Wei Ye Mengfan Dong Ming Yan Ji Zhang Fei Huang Shikun Zhang MLLM 89 2 0 17 Nov 2024
Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model Ben Koska Mojmír Horváth MoE 37 1 0 08 Nov 2024
M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding Jaemin Cho Debanjan Mahata Ozan Irsoy Yujie He Mohit Bansal VLM 29 9 0 07 Nov 2024
Both Text and Images Leaked! A Systematic Analysis of Multimodal LLM Data Contamination D. Song Sicheng Lai Shunian Chen Lichao Sun Benyou Wang 148 0 0 06 Nov 2024
KptLLM: Unveiling the Power of Large Language Model for Keypoint Comprehension Jie-jin Yang Wang Zeng Sheng Jin Lumin Xu Wentao Liu Chen Qian Ruimao Zhang MLLM 65 2 0 04 Nov 2024