MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning

21 December 2022

Papers citing "MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning"

50 / 92 papers shown

Title
Rethinking Memory in AI: Taxonomy, Operations, Topics, and Future Directions Yiming Du Wenyu Huang Danna Zheng Zhaowei Wang Sébastien Montella Mirella Lapata Kam-Fai Wong Jeff Z. Pan KELM MU 78 2 0 01 May 2025
Fast-Slow-Thinking: Complex Task Solving with Large Language Models Yiliu Sun Yanfang Zhang Zicheng Zhao Sheng Wan Dacheng Tao Chen Gong LRM 33 0 0 11 Apr 2025
MM-IFEngine: Towards Multimodal Instruction Following Shengyuan Ding Shenxi Wu Xiangyu Zhao Yuhang Zang Haodong Duan Xiaoyi Dong Pan Zhang Y. Cao D. Lin Jiaqi Wang OffRL 56 1 0 10 Apr 2025
FortisAVQA and MAVEN: a Benchmark Dataset and Debiasing Framework for Robust Multimodal Reasoning Jie Ma Zhitao Gao Qi Chai J. Liu P. Wang Jing Tao Zhou Su 48 0 0 01 Apr 2025
When Words Outperform Vision: VLMs Can Self-Improve Via Text-Only Training For Human-Centered Decision Making Zhe Hu Jing Li Yu Yin VLM 63 0 0 21 Mar 2025
Picking the Cream of the Crop: Visual-Centric Data Selection with Collaborative Agents Zhenyu Liu Yunxin Li Baotian Hu Wenhan Luo Yaowei Wang Min-Ling Zhang 62 0 0 27 Feb 2025
VideoSAVi: Self-Aligned Video Language Models without Human Supervision Yogesh Kulkarni Pooyan Fazli VLM 100 2 0 01 Dec 2024
MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models Jianhong Tu Zhuohao Ni Nicholas Crispino Zihao Yu Michael Bendersky ... Ruoxi Jia Xin Liu Lingjuan Lyu Dawn Song Chenguang Wang VLM MLLM 49 0 0 15 Nov 2024
$Adapt-$\infty$: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection$ Adapt- $\infty$ : Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection A. Maharana Jaehong Yoon Tianlong Chen Mohit Bansal 31 0 0 14 Oct 2024
M $^2$ PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning Taowen Wang Yiyang Liu James Liang Junhan Zhao Yiming Cui ... Zenglin Xu Cheng Han Lifu Huang Qifan Wang Dongfang Liu MLLM VLM LRM 22 16 0 24 Sep 2024
From Linguistic Giants to Sensory Maestros: A Survey on Cross-Modal Reasoning with Large Language Models Shengsheng Qian Zuyi Zhou Dizhan Xue Bing Wang Changsheng Xu LRM 36 1 0 19 Sep 2024
MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity Yangzhou Liu Yue Cao Zhangwei Gao Weiyun Wang Zhe Chen ... Lewei Lu Xizhou Zhu Tong Lu Yu Qiao Jifeng Dai VLM MLLM 49 23 0 22 Jul 2024
Advancing Chart Question Answering with Robust Chart Component Recognition Hanwen Zheng Sijia Wang Chris Thomas Lifu Huang 37 1 0 19 Jul 2024
HEMM: Holistic Evaluation of Multimodal Foundation Models Paul Pu Liang Akshay Goindani Talha Chafekar Leena Mathur Haofei Yu Ruslan Salakhutdinov Louis-Philippe Morency 38 10 0 03 Jul 2024
MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment Jihao Liu Xin Huang Jinliang Zheng Boxiao Liu Jia Wang Osamu Yoshie Yu Liu Hongsheng Li MLLM SyDa 30 3 0 28 Jun 2024
Holistic Evaluation for Interleaved Text-and-Image Generation Minqian Liu Zhiyang Xu Zihao Lin Trevor Ashby Joy Rimchala Jiaxin Zhang Lifu Huang EGVM 36 7 0 20 Jun 2024
Multimodal Table Understanding Mingyu Zheng Xinwei Feng Q. Si Qiaoqiao She Zheng-Shen Lin Wenbin Jiang Weiping Wang LMTD VLM 42 14 0 12 Jun 2024
An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Models Xiongtao Zhou Jie He Yuhua Ke Guangyao Zhu Víctor Gutiérrez-Basulto Jeff Z. Pan 40 11 0 07 Jun 2024
Libra: Building Decoupled Vision System on Large Language Models Yifan Xu Xiaoshan Yang Y. Song Changsheng Xu MLLM VLM 38 6 0 16 May 2024
EmoVIT: Revolutionizing Emotion Insights with Visual Instruction Tuning Hongxia Xie Chu-Jun Peng Yu-Wen Tseng Hung-Jen Chen Chan-Feng Hsu Hong-Han Shuai Wen-Huang Cheng 44 15 0 25 Apr 2024
What Makes Multimodal In-Context Learning Work? Folco Bertini Baldassini Mustafa Shukor Matthieu Cord Laure Soulier Benjamin Piwowarski 34 18 0 24 Apr 2024
Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models Jesse Atuhurra Iqra Ali Tatsuya Hiraoka Hidetaka Kamigaito Tomoya Iwakura Taro Watanabe 38 1 0 29 Mar 2024
UniCode: Learning a Unified Codebook for Multimodal Large Language Models Sipeng Zheng Bohan Zhou Yicheng Feng Ye Wang Zongqing Lu VLM MLLM 40 7 0 14 Mar 2024
Peacock: A Family of Arabic Multimodal Large Language Models and Benchmarks Fakhraddin Alwajih El Moatez Billah Nagoudi Gagan Bhatia Abdelrahman Mohamed Muhammad Abdul-Mageed VLM LRM 27 11 0 01 Mar 2024
Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Models Lei Li Yuqi Wang Runxin Xu Peiyi Wang Xiachong Feng Lingpeng Kong Qi Liu 37 51 0 01 Mar 2024
ShapeLLM: Universal 3D Object Understanding for Embodied Interaction Zekun Qi Runpei Dong Shaochen Zhang Haoran Geng Chunrui Han Zheng Ge Li Yi Kaisheng Ma 41 51 0 27 Feb 2024
VCD: Knowledge Base Guided Visual Commonsense Discovery in Images Xiangqing Shen Yurun Song Siwei Wu Rui Xia 33 6 0 27 Feb 2024
LSTP: Language-guided Spatial-Temporal Prompt Learning for Long-form Video-Text Understanding Yuxuan Wang Yueqian Wang Pengfei Wu Jianxin Liang Dongyan Zhao Zilong Zheng VLM 28 9 0 25 Feb 2024
Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions Akash Ghosh Arkadeep Acharya Sriparna Saha Vinija Jain Aman Chadha VLM 51 25 0 20 Feb 2024
Scaffolding Coordinates to Promote Vision-Language Coordination in Large Multi-Modal Models Xuanyu Lei Zonghan Yang Xinrui Chen Peng Li Yang Liu MLLM LRM 40 30 0 19 Feb 2024
eCeLLM: Generalizing Large Language Models for E-commerce from Large-scale, High-quality Instruction Data B. Peng Xinyi Ling Ziru Chen Huan Sun Xia Ning ELM 29 16 0 13 Feb 2024
Read to Play (R2-Play): Decision Transformer with Multimodal Game Instruction Yonggang Jin Ge Zhang Hao Zhao Tianyu Zheng Jiawei Guo Liuyu Xiang Shawn Yue Stephen W. Huang Zhaofeng He Jie Fu OffRL 29 4 0 06 Feb 2024
InstructDoc: A Dataset for Zero-Shot Generalization of Visual Document Understanding with Instructions Ryota Tanaka Taichi Iki Kyosuke Nishida Kuniko Saito Jun Suzuki VLM 18 23 0 24 Jan 2024
Supervised Fine-tuning in turn Improves Visual Foundation Models Xiaohu Jiang Yixiao Ge Yuying Ge Dachuan Shi Chun Yuan Ying Shan VLM CLIP 40 8 0 18 Jan 2024
Towards Truly Zero-shot Compositional Visual Reasoning with LLMs as Programmers Aleksandar Stanić Sergi Caelles Michael Tschannen LRM VLM 25 9 0 03 Jan 2024
Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey Jiaxing Huang Jingyi Zhang Kai Jiang Han Qiu Shijian Lu 32 22 0 27 Dec 2023
Lyrics: Boosting Fine-grained Language-Vision Alignment and Comprehension via Semantic-aware Visual Objects Junyu Lu Ruyi Gan Di Zhang Xiaojun Wu Ziwei Wu Renliang Sun Jiaxing Zhang Pingjian Zhang Yan Song MLLM VLM 20 15 0 08 Dec 2023
Digital Life Project: Autonomous 3D Characters with Social Intelligence Zhongang Cai Jian-Dong Jiang Zhongfei Qing Xinying Guo Mingyuan Zhang ... Yukun Wei Xiaogang Wang Chen Change Loy Lei Yang Ziwei Liu VGen 30 24 0 07 Dec 2023
MUFFIN: Curating Multi-Faceted Instructions for Improving Instruction-Following Renze Lou Kai Zhang Jian Xie Yuxuan Sun Janice Ahn Hanzi Xu Yu Su Wenpeng Yin 29 26 0 05 Dec 2023
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding Shuhuai Ren Linli Yao Shicheng Li Xu Sun Lu Hou VLM MLLM 23 174 0 04 Dec 2023
Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites Lei Wang Jiabang He Shenshen Li Ning Liu Ee-Peng Lim MLLM 27 38 0 04 Dec 2023
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers Cong Wei Yang Chen Haonan Chen Hexiang Hu Ge Zhang Jie Fu Alan Ritter Wenhu Chen 40 50 0 28 Nov 2023
DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback Yangyi Chen Karan Sikka Michael Cogswell Heng Ji Ajay Divakaran 26 58 0 16 Nov 2023
Vision-Language Instruction Tuning: A Review and Analysis Chen Li Yixiao Ge Dian Li Ying Shan VLM 33 12 0 14 Nov 2023
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models Zhang Li Biao Yang Qiang Liu Zhiyin Ma Shuo Zhang Jingxu Yang Yabo Sun Yuliang Liu Xiang Bai MLLM 36 241 0 11 Nov 2023
What Makes for Good Visual Instructions? Synthesizing Complex Visual Reasoning Instructions for Visual Instruction Tuning Yifan Du Hangyu Guo Kun Zhou Wayne Xin Zhao Jinpeng Wang Chuyuan Wang Mingchen Cai Ruihua Song Ji-Rong Wen VLM MLLM LRM 61 22 0 02 Nov 2023
Ziya-Visual: Bilingual Large Vision-Language Model via Multi-Task Instruction Tuning Junyu Lu Di Zhang Xiaojun Wu Xinyu Gao Ruyi Gan Jiaxing Zhang Yan Song Pingjian Zhang VLM MLLM 17 7 0 12 Oct 2023
MULTISCRIPT: Multimodal Script Learning for Supporting Open Domain Everyday Tasks Jingyuan Qi Minqian Liu Ying Shen Zhiyang Xu Lifu Huang LRM VGen 29 2 0 08 Oct 2023
ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks Zejun Li Ye Wang Mengfei Du Qingwen Liu Binhao Wu ... Zhihao Fan Jie Fu Jingjing Chen Xuanjing Huang Zhongyu Wei 27 13 0 04 Oct 2023
Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning Mustafa Shukor Alexandre Ramé Corentin Dancette Matthieu Cord LRM MLLM 38 20 0 01 Oct 2023