Title
LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images? Maoyuan Ye Jing Zhang Juhua Liu Bo Du Dacheng Tao LRM 160 0 0 18 May 2025
DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding Z. F. Wu Xiaokang Chen Zizheng Pan Xianglong Liu Wen Liu ... Xingkai Yu Haowei Zhang Liang Zhao Yijiao Wang Chong Ruan MLLM VLM MoE 184 158 0 13 Dec 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 129 92 1 15 Nov 2024
Locality Alignment Improves Vision-Language Models Ian Covert Tony Sun James Zou Tatsunori Hashimoto VLM 245 7 0 14 Oct 2024
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning Haotian Zhang Mingfei Gao Zhe Gan Philipp Dufter Nina Wenzel ... Haoxuan You Zirui Wang Afshin Dehghan Peter Grasch Yinfei Yang VLM MLLM 108 41 1 30 Sep 2024
mPLUG-DocOwl2: High-resolution Compressing for OCR-free Multi-page Document Understanding Anwen Hu Haiyang Xu Liang Zhang Jiabo Ye Ming Yan Ji Zhang Qin Jin Fei Huang Jingren Zhou VLM 81 37 0 05 Sep 2024
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders Min Shi Fuxiao Liu Shihao Wang Shijia Liao Subhashree Radhakrishnan ... Andrew Tao Andrew Tao Zhiding Yu Guilin Liu Guilin Liu MLLM 127 68 0 28 Aug 2024
DocLayLLM: An Efficient Multi-modal Extension of Large Language Models for Text-rich Document Understanding Wenhui Liao Jiapeng Wang Hongliang Li Chengyu Wang Jun Huang Lianwen Jin 122 0 0 27 Aug 2024
Building and better understanding vision-language models: insights and future directions Hugo Laurençon Andrés Marafioti Victor Sanh Léo Tronchon VLM 112 77 0 22 Aug 2024
LLaVA-OneVision: Easy Visual Task Transfer Bo Li Yuanhan Zhang Dong Guo Renrui Zhang Feng Li Hao Zhang Kaichen Zhang Yanwei Li Ziwei Liu Chunyuan Li MLLM SyDa VLM 121 867 0 06 Aug 2024
Mini-Monkey: Multi-Scale Adaptive Cropping for Multimodal Large Language Models Mingxin Huang Yuliang Liu Dingkang Liang Lianwen Jin Xiang Bai 91 14 0 04 Aug 2024
Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding Renshan Zhang Yibo Lyu Rui Shao Gongwei Chen Weili Guan Liqiang Nie 72 10 0 19 Jul 2024
PosFormer: Recognizing Complex Handwritten Mathematical Expression with Position Forest Transformer Tongkun Guan Chengyu Lin Wei Shen Xiaokang Yang 72 6 0 10 Jul 2024
TokenPacker: Efficient Visual Projector for Multimodal LLM Wentong Li Yuqian Yuan Jian Liu Dongqi Tang Song Wang Jie Qin Jianke Zhu Lei Zhang MLLM 113 67 0 02 Jul 2024
A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding Jinghui Lu Haiyang Yu Yanjie Wang Yongjie Ye Jingqun Tang ... Qi Liu Hao Feng Han Wang Hao Liu Can Huang 138 23 0 02 Jul 2024
DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming Jiaxin Zhang Wentao Yang Songxuan Lai Zecheng Xie Lianwen Jin 75 21 0 27 Jun 2024
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 132 378 0 24 Jun 2024
DocGenome: An Open Large-scale Scientific Document Benchmark for Training and Testing Multi-modal Large Language Models Renqiu Xia Song Mao Xiangchao Yan Hongbin Zhou Bo Zhang ... Yongwei Wang Bin Wang Junchi Yan Fei Wu Yu Qiao 87 12 0 17 Jun 2024
What matters when building vision-language models? Hugo Laurençon Léo Tronchon Matthieu Cord Victor Sanh VLM 98 177 0 03 May 2024
How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites Zhe Chen Weiyun Wang Hao Tian Shenglong Ye Zhangwei Gao ... Tong Lu Dahua Lin Yu Qiao Jifeng Dai Wenhai Wang MLLM VLM 115 642 0 25 Apr 2024
HRVDA: High-Resolution Visual Document Assistant Chaohu Liu Kun Yin Haoyu Cao Xinghua Jiang Xin Li Yinsong Liu Deqiang Jiang Xing Sun Linli Xu VLM 87 26 0 10 Apr 2024
LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding Chuwei Luo Yufan Shen Zhaoqing Zhu Qi Zheng Zhi Yu Cong Yao 104 49 0 08 Apr 2024
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding Anwen Hu Haiyang Xu Jiabo Ye Mingshi Yan Liang Zhang ... Chen Li Ji Zhang Qin Jin Fei Huang Jingren Zhou VLM 102 124 0 19 Mar 2024
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training Brandon McKinzie Zhe Gan J. Fauconnier Sam Dodge Bowen Zhang ... Zirui Wang Ruoming Pang Peter Grasch Alexander Toshev Yinfei Yang MLLM 92 208 0 14 Mar 2024
Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset Hugo Laurençon Léo Tronchon Victor Sanh VLM 81 43 0 14 Mar 2024
MoAI: Mixture of All Intelligence for Large Language and Vision Models Byung-Kwan Lee Beomchan Park Chae Won Kim Yonghyun Ro MLLM VLM 111 23 0 12 Mar 2024
TextMonkey: An OCR-Free Large Multimodal Model for Understanding Document Yuliang Liu Biao Yang Qiang Liu Zhang Li Zhiyin Ma Shuo Zhang Xiang Bai MLLM VLM 108 107 0 07 Mar 2024
OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web Raghav Kapoor Y. Butala M. Russak Jing Yu Koh Kiran Kamble Waseem Alshikh Ruslan Salakhutdinov LLMAG 118 57 0 27 Feb 2024
Hi-SAM: Marrying Segment Anything Model for Hierarchical Text Segmentation Maoyuan Ye Jing Zhang Juhua Liu Chenyu Liu Baocai Yin Cong Liu Bo Du Dacheng Tao VLM 90 15 0 31 Jan 2024
MouSi: Poly-Visual-Expert Vision-Language Models Xiaoran Fan Tao Ji Changhao Jiang Shuo Li Senjie Jin ... Qi Zhang Xipeng Qiu Xuanjing Huang Zuxuan Wu Yunchun Jiang VLM 46 17 0 30 Jan 2024
InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions Ryota Tanaka Taichi Iki Kyosuke Nishida Kuniko Saito Jun Suzuki VLM 63 23 0 24 Jan 2024
SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents Kanzhi Cheng Qiushi Sun Yougang Chu Fangzhi Xu Yantao Li Jianbing Zhang Zhiyong Wu LLMAG 255 189 0 17 Jan 2024
DocLLM: A layout-aware generative language model for multimodal document understanding Dongsheng Wang Natraj Raman Mathieu Sibue Zhiqiang Ma Petr Babkin Simerjot Kaur Yulong Pei Armineh Nourbakhsh Xiaomo Liu VLM 82 61 0 31 Dec 2023
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 260 1,215 0 21 Dec 2023
Bridging Synthetic and Real Worlds for Pre-training Scene Text Detectors Tongkun Guan Wei Shen Xuehang Yang Xuehui Wang Xiaokang Yang 88 7 0 08 Dec 2023
DocPedia: Unleashing the Power of Large Multimodal Model in the Frequency Domain for Versatile Document Understanding Hao Feng Qi Liu Hao Liu Wen-gang Zhou Houqiang Li Can Huang VLM 91 67 0 20 Nov 2023
SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models Ziyi Lin Chris Liu Renrui Zhang Peng Gao Longtian Qiu ... Siyuan Huang Yichi Zhang Xuming He Hongsheng Li Yu Qiao MLLM VLM 87 230 0 13 Nov 2023
SpectralGPT: Spectral Remote Sensing Foundation Model Danfeng Hong Bing Zhang Xuyang Li Yuxuan Li Chenyu Li ... Xiuping Jia Antonio J. Plaza Paolo Gamba J. Benediktsson J. Chanussot 99 421 0 13 Nov 2023
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models Zhang Li Biao Yang Qiang Liu Zhiyin Ma Shuo Zhang Jingxu Yang Yabo Sun Yuliang Liu Xiang Bai MLLM 101 276 0 11 Nov 2023
UReader: Universal OCR-free Visually-situated Language Understanding with Multimodal Large Language Model Jiabo Ye Anwen Hu Haiyang Xu Qinghao Ye Mingshi Yan ... Ji Zhang Qin Jin Liang He Xin Lin Feiyan Huang VLM MLLM 191 97 0 08 Oct 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 168 2,825 0 05 Oct 2023
Kosmos-2.5: A Multimodal Literate Model Tengchao Lv Yupan Huang Jingye Chen Lei Cui Shuming Ma ... Weiyao Luo Shaoxiang Wu Guoxin Wang Cha Zhang Furu Wei VLM MLLM 103 66 0 20 Sep 2023
UniDoc: A Universal Large Multimodal Model for Simultaneous Text Detection, Recognition, Spotting and Understanding Hao Feng Zijian Wang Jingqun Tang Jinghui Lu Wen-gang Zhou Houqiang Li Can Huang MLLM VLM 119 51 0 19 Aug 2023
LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding Yanzhe Zhang Ruiyi Zhang Jiuxiang Gu Yufan Zhou Nedim Lipka Diyi Yang Tongfei Sun VLM MLLM 83 237 0 29 Jun 2023
Faster Segment Anything: Towards Lightweight SAM for Mobile Applications Chaoning Zhang Dongshen Han Yu Qiao Jung Uk Kim Sung-Ho Bae Seungkyu Lee Choong Seon Hong VLM 98 359 0 25 Jun 2023
Mind2Web: Towards a Generalist Agent for the Web Xiang Deng Yu Gu Boyuan Zheng Shijie Chen Samuel Stevens Boshi Wang Huan Sun Yu-Chuan Su LLMAG 105 486 0 09 Jun 2023
Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena Lianmin Zheng Wei-Lin Chiang Ying Sheng Siyuan Zhuang Zhanghao Wu ... Dacheng Li Eric Xing Haotong Zhang Joseph E. Gonzalez Ion Stoica ALM OSLM ELM 438 4,444 0 09 Jun 2023
PaLI-X: On Scaling up a Multilingual Vision and Language Model Xi Chen Josip Djolonga Piotr Padlewski Basil Mustafa Soravit Changpinyo ... Mojtaba Seyedhosseini A. Angelova Xiaohua Zhai N. Houlsby Radu Soricut VLM 132 204 0 29 May 2023
CCpdf: Building a High Quality Corpus for Visually Rich Documents from Web Crawl Data M. Turski Tomasz Stanislawek Karol Kaczmarek Pawel Dyda Filip Graliñski 72 12 0 28 Apr 2023
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models Deyao Zhu Jun Chen Xiaoqian Shen Xiang Li Mohamed Elhoseiny VLM MLLM 165 2,069 0 20 Apr 2023