IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities

23 August 2024

Papers citing "IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities"

34 / 34 papers shown

Title
FG-CLIP: Fine-Grained Visual and Textual Alignment Chunyu Xie Bin Wang Fanjing Kong Jincheng Li Dawei Liang Gengshen Zhang Dawei Leng Yuhui Yin CLIP VLM 99 0 0 08 May 2025
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs Shengbang Tong Ellis L Brown Penghao Wu Sanghyun Woo Manoj Middepogu ... Xichen Pan Austin Wang Rob Fergus Yann LeCun Saining Xie 3DV MLLM 80 321 0 24 Jun 2024
DeepSeek-VL: Towards Real-World Vision-Language Understanding Haoyu Lu Wen Liu Bo Zhang Bing-Li Wang Kai Dong ... Yaofeng Sun Chengqi Deng Hanwei Xu Zhenda Xie Chong Ruan VLM 71 333 0 08 Mar 2024
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 211 1,036 0 21 Dec 2023
Honeybee: Locality-enhanced Projector for Multimodal LLM Junbum Cha Wooyoung Kang Jonghwan Mun Byungseok Roh MLLM 53 124 0 11 Dec 2023
MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI Xiang Yue Yuansheng Ni Kai Zhang Tianyu Zheng Ruoqi Liu ... Yibo Liu Wenhao Huang Huan Sun Yu-Chuan Su Wenhu Chen OSLM ELM VLM 148 833 0 27 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 158 395 0 07 Nov 2023
Ferret: Refer and Ground Anything Anywhere at Any Granularity Haoxuan You Haotian Zhang Zhe Gan Xianzhi Du Bowen Zhang Zirui Wang Liangliang Cao Shih-Fu Chang Yinfei Yang ObjD MLLM VLM 65 314 0 11 Oct 2023
Improved Baselines with Visual Instruction Tuning Haotian Liu Chunyuan Li Yuheng Li Yong Jae Lee VLM MLLM 89 2,593 0 05 Oct 2023
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs Shiyu Xuan Qingpei Guo Ming Yang Shiliang Zhang MLLM ObjD 37 40 0 01 Oct 2023
What Makes Good Open-Vocabulary Detector: A Disassembling Perspective Jincheng Li Chunyu Xie Xiaoyu Wu Bin Wang Dawei Leng VLM ObjD 65 3 0 01 Sep 2023
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning Tri Dao LRM 69 1,221 0 17 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Jiaqi Wang Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 44 992 0 12 Jul 2023
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic Ke Chen Zhao Zhang Weili Zeng Richong Zhang Feng Zhu Rui Zhao ObjD 62 622 0 27 Jun 2023
Kosmos-2: Grounding Multimodal Large Language Models to the World Zhiliang Peng Wenhui Wang Li Dong Y. Hao Shaohan Huang Shuming Ma Furu Wei MLLM ObjD VLM 78 735 0 26 Jun 2023
C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models Yuzhen Huang Yuzhuo Bai Zhihao Zhu Junlei Zhang Jinghan Zhang ... Yikai Zhang Jiayi Lei Yao Fu Maosong Sun Junxian He ELM LRM 51 528 0 15 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 75 2,017 0 11 May 2023
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model Peng Gao Jiaming Han Renrui Zhang Ziyi Lin Shijie Geng ... Pan Lu Conghui He Xiangyu Yue Hongsheng Li Yu Qiao MLLM 67 572 0 28 Apr 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 240 931 0 27 Apr 2023
Sigmoid Loss for Language Image Pre-Training Xiaohua Zhai Basil Mustafa Alexander Kolesnikov Lucas Beyer CLIP VLM 83 1,076 0 27 Mar 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 228 1,188 0 20 Sep 2022
CCMB: A Large-scale Chinese Cross-modal Benchmark Chunyu Xie Heng Cai Jincheng Li Fanjing Kong Xiaoyu Wu ... Xiangzheng Zhang Dawei Leng Baochang Zhang Xiangyang Ji Yafeng Deng MLLM VLM 31 10 0 08 May 2022
OPT: Open Pre-trained Transformer Language Models Susan Zhang Stephen Roller Naman Goyal Mikel Artetxe Moya Chen ... Daniel Simig Punit Singh Koura Anjali Sridhar Tianlu Wang Luke Zettlemoyer VLM OSLM AI4CE 282 3,583 0 02 May 2022
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework Peng Wang An Yang Rui Men Junyang Lin Shuai Bai Zhikang Li Jianxin Ma Chang Zhou Jingren Zhou Hongxia Yang MLLM ObjD 120 865 0 07 Feb 2022
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 681 28,659 0 26 Feb 2021
Measuring Massive Multitask Language Understanding Dan Hendrycks Collin Burns Steven Basart Andy Zou Mantas Mazeika D. Song Jacob Steinhardt ELM RALM 137 4,222 0 07 Sep 2020
DocVQA: A Dataset for VQA on Document Images Minesh Mathew Dimosthenis Karatzas C. V. Jawahar 97 700 0 01 Jul 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 500 41,106 0 28 May 2020
ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Samyam Rajbhandari Jeff Rasley Olatunji Ruwase Yuxiong He ALM AI4CE 65 852 0 04 Oct 2019
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin Ming-Wei Chang Kenton Lee Kristina Toutanova VLM SSL SSeg 966 93,936 0 11 Oct 2018
Modeling Context in Referring Expressions Licheng Yu Patrick Poirson Shan Yang Alexander C. Berg Tamara L. Berg 108 1,250 0 31 Jul 2016
Generation and Comprehension of Unambiguous Object Descriptions Junhua Mao Jonathan Huang Alexander Toshev Oana-Maria Camburu Alan Yuille Kevin Patrick Murphy ObjD 92 1,335 0 07 Nov 2015
Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models Bryan A. Plummer Liwei Wang Christopher M. Cervantes Juan C. Caicedo Julia Hockenmaier Svetlana Lazebnik 177 2,033 0 19 May 2015
Microsoft COCO: Common Objects in Context Nayeon Lee Michael Maire Serge J. Belongie Lubomir Bourdev Ross B. Girshick James Hays Pietro Perona Deva Ramanan C. L. Zitnick Piotr Dollár ObjD 272 43,290 0 01 May 2014