Visual Instruction Tuning

17 April 2023

Papers citing "Visual Instruction Tuning"

50 / 3,278 papers shown

Title
Versatile Backdoor Attack with Visible, Semantic, Sample-Specific, and Compatible Triggers Ke Xu Hongrui Chen Zihao Zhu Li Liu Baoyuan Wu DiffM 38 11 0 01 Jun 2023
Adapting Pre-trained Language Models to Vision-Language Tasks via Dynamic Visual Prompting Shubin Huang Qiong Wu Yiyi Zhou Weijie Chen Rongsheng Zhang Xiaoshuai Sun Rongrong Ji VLM VPVLM LRM 16 0 0 01 Jun 2023
GPT4Image: Large Pre-trained Models Help Vision Models Learn Better on Perception Task Ning Ding Yehui Tang Zhongqian Fu Chaoting Xu Kai Han Yunhe Wang MLLM VLM 37 2 0 01 Jun 2023
Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Models Sivan Doveh Assaf Arbelle Sivan Harary Roei Herzig Donghyun Kim ... Yikang Shen Raja Giryes Rogerio Feris S. Ullman Leonid Karlinsky VLM CoGe 62 53 0 31 May 2023
Contextual Object Detection with Multimodal Large Language Models Yuhang Zang Wei Li Jun Han Kaiyang Zhou Chen Change Loy ObjD VLM MLLM 47 78 0 29 May 2023
GlyphControl: Glyph Conditional Control for Visual Text Generation Yukang Yang Dongnan Gui Yuhui Yuan Weicong Liang Haisong Ding Hang-Rui Hu Kai Chen DiffM 38 78 0 29 May 2023
CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformers Dachuan Shi Chaofan Tao Anyi Rao Zhendong Yang Chun Yuan Jiaqi Wang VLM 39 22 0 27 May 2023
Generating Images with Multimodal Language Models Jing Yu Koh Daniel Fried Ruslan Salakhutdinov MLLM 44 243 0 26 May 2023
On Evaluating Adversarial Robustness of Large Vision-Language Models Yunqing Zhao Tianyu Pang Chao Du Xiao Yang Chongxuan Li Ngai-man Cheung Min Lin VLM AAML MLLM 35 166 0 26 May 2023
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst Zijia Zhao Longteng Guo Tongtian Yue Si-Qing Chen Shuai Shao Xinxin Zhu Zehuan Yuan Jing Liu MLLM 42 53 0 25 May 2023
Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback Yiqi Lin Hao Wu Ruichen Wang H. Lu Xiaodong Lin Hui Xiong Lin Wang 3DV 48 12 0 25 May 2023
The False Promise of Imitating Proprietary LLMs Arnav Gudibande Eric Wallace Charles Burton Snell Xinyang Geng Hao Liu Pieter Abbeel Sergey Levine Dawn Song ALM 44 199 0 25 May 2023
PandaGPT: One Model To Instruction-Follow Them All Yixuan Su Tian Lan Huayang Li Jialu Xu Yan Wang Deng Cai MLLM 47 278 0 25 May 2023
Rethinking the Evaluation Protocol of Domain Generalization Han Yu Xingxuan Zhang Renzhe Xu Jiashuo Liu Yue He Peng Cui OOD 39 7 0 24 May 2023
Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models Geewook Kim Hodong Lee D. Kim Haeji Jung S. Park Yoon Kim Sangdoo Yun Taeho Kil Bado Lee Seunghyun Park VLM 53 4 0 24 May 2023
Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Models Gen Luo Yiyi Zhou Tianhe Ren Shen Chen Xiaoshuai Sun Rongrong Ji VLM MLLM 31 91 0 24 May 2023
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought Yao Mu Qinglong Zhang Mengkang Hu Wen Wang Mingyu Ding Jun Jin Bin Wang Jifeng Dai Yu Qiao Ping Luo LM&Ro LRM 39 224 0 24 May 2023
Mixture-of-Experts Meets Instruction Tuning:A Winning Combination for Large Language Models Sheng Shen Le Hou Yan-Quan Zhou Nan Du Shayne Longpre ... Vincent Zhao Hongkun Yu Kurt Keutzer Trevor Darrell Denny Zhou ALM MoE 45 54 0 24 May 2023
IdealGPT: Iteratively Decomposing Vision and Language Reasoning via Large Language Models Haoxuan You Rui Sun Zhecan Wang Long Chen Gengyu Wang Hammad A. Ayyubi Kai-Wei Chang Shih-Fu Chang VLM MLLM LRM 58 43 0 24 May 2023
CREATOR: Tool Creation for Disentangling Abstract and Concrete Reasoning of Large Language Models Cheng Qian Chi Han Yi R. Fung Yujia Qin Zhiyuan Liu Heng Ji LRM 22 30 0 23 May 2023
ReSee: Responding through Seeing Fine-grained Visual Knowledge in Open-domain Dialogue Haoqin Tu Yitong Li Fei Mi Zhongliang Yang 46 4 0 23 May 2023
Training Diffusion Models with Reinforcement Learning Kevin Black Michael Janner Yilun Du Ilya Kostrikov Sergey Levine EGVM 46 320 0 22 May 2023
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback Yann Dubois Xuechen Li Rohan Taori Tianyi Zhang Ishaan Gulrajani Jimmy Ba Carlos Guestrin Percy Liang Tatsunori B. Hashimoto ALM 47 549 0 22 May 2023
TheoremQA: A Theorem-driven Question Answering dataset Wenhu Chen Ming Yin Max W.F. Ku Pan Lu Yixin Wan Xueguang Ma Jianyu Xu Xinyi Wang Tony Xia AIMat 38 125 0 21 May 2023
What Makes for Good Visual Tokenizers for Large Language Models? Guangzhi Wang Yixiao Ge Xiaohan Ding Mohan S. Kankanhalli Ying Shan MLLM VLM 33 39 0 20 May 2023
Multimodal Web Navigation with Instruction-Finetuned Foundation Models Hiroki Furuta Kuang-Huei Lee Ofir Nachum Yutaka Matsuo Aleksandra Faust S. Gu Izzeddin Gur LM&Ro 44 93 0 19 May 2023
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks Wen Wang Zhe Chen Xiaokang Chen Jiannan Wu Xizhou Zhu ... Ping Luo Tong Lu Jie Zhou Yu Qiao Jifeng Dai MLLM VLM 38 464 0 18 May 2023
Going Denser with Open-Vocabulary Part Segmentation Pei Sun Shoufa Chen Chenchen Zhu Fanyi Xiao Ping Luo Saining Xie Zhicheng Yan ObjD VLM 27 46 0 18 May 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 53 116 0 18 May 2023
DrugChat: Towards Enabling ChatGPT-Like Capabilities on Drug Molecule Graphs Youwei Liang Ruiyi Zhang Li Zhang Pengtao Xie LM&MA GNN 21 48 0 18 May 2023
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities Dong Zhang Shimin Li Xin Zhang Jun Zhan Pengyu Wang Yaqian Zhou Xipeng Qiu AuLLM MLLM 62 302 0 18 May 2023
Listen, Think, and Understand Yuan Gong Hongyin Luo Alexander H. Liu Leonid Karlinsky James R. Glass ELM MLLM LRM 45 141 0 18 May 2023
Discffusion: Discriminative Diffusion Models as Few-shot Vision and Language Learners Xuehai He Weixi Feng Tsu-Jui Fu Varun Jampani Arjun Reddy Akula P. Narayana Sugato Basu William Yang Wang Xinze Wang DiffM 62 7 0 18 May 2023
Evaluating Object Hallucination in Large Vision-Language Models Yifan Li Yifan Du Kun Zhou Jinpeng Wang Wayne Xin Zhao Ji-Rong Wen MLLM LRM 134 707 0 17 May 2023
On the Hidden Mystery of OCR in Large Multimodal Models Yuliang Liu Zhang Li Mingxin Huang Chunyuan Li Dezhi Peng Mingyu Liu Lianwen Jin Xiang Bai VLM MLLM 39 55 0 13 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 19 1,919 0 11 May 2023
VideoChat: Chat-Centric Video Understanding Kunchang Li Yinan He Yi Wang Yizhuo Li Wen Wang Ping Luo Yali Wang Limin Wang Yu Qiao MLLM 69 534 0 10 May 2023
Vision-Language Models in Remote Sensing: Current Progress and Future Trends Xiang Li Congcong Wen Yuan Hu Zhenghang Yuan Xiao Xiang Zhu VLM 32 75 0 09 May 2023
InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language Zhaoyang Liu Yinan He Wenhai Wang Weiyun Wang Yi Wang ... Yali Wang Limin Wang Ping Luo Jifeng Dai Yu Qiao LRM MLLM 47 79 0 09 May 2023
MultiModal-GPT: A Vision and Language Model for Dialogue with Humans T. Gong Chengqi Lyu Shilong Zhang Yudong Wang Miao Zheng Qianmengke Zhao Kuikun Liu Wenwei Zhang Ping Luo Kai-xiang Chen MLLM 34 254 0 08 May 2023
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages Feilong Chen Minglun Han Haozhi Zhao Qingyang Zhang Jing Shi Shuang Xu Bo Xu MLLM 46 116 0 07 May 2023
Otter: A Multi-Modal Model with In-Context Instruction Tuning Yue Liu Yuanhan Zhang Liangyu Chen Jinghao Wang Jingkang Yang Ziwei Liu MLLM 48 505 0 05 May 2023
LMEye: An Interactive Perception Network for Large Language Models Yunxin Li Baotian Hu Xinyu Chen Lin Ma Yong-mei Xu Min Zhang MLLM VLM 33 24 0 05 May 2023
T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning via Mixed Large Language Model Signals for Science Question Answering Lei Wang Yilang Hu Jiabang He Xingdong Xu Ning Liu Hui-juan Liu Hengtao Shen LRM MLLM 34 41 0 05 May 2023
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision Zhiqing Sun Songlin Yang Qinhong Zhou Hongxin Zhang Zhenfang Chen David D. Cox Yiming Yang Chuang Gan SyDa ALM 27 315 0 04 May 2023
Visual Transformation Telling Wanqing Cui Mustafa Nasir-Moin Yanyan Lan Viola J. Chen Jiafeng Guo Xueqi Cheng LRM 67 1 0 03 May 2023
Instruction-ViT: Multi-Modal Prompts for Instruction Learning in ViT Zhe Xiao Yuzhong Chen Lu Zhang Jun Yao Zihao Wu ... Yixuan Yuan Dinggang Shen Dajiang Zhu Tianming Liu Xi Jiang VLM MLLM 78 17 0 29 Apr 2023
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model Peng Gao Jiaming Han Renrui Zhang Ziyi Lin Shijie Geng ... Pan Lu Conghui He Xiangyu Yue Hongsheng Li Yu Qiao MLLM 47 560 0 28 Apr 2023
WizardLM: Empowering Large Language Models to Follow Complex Instructions Can Xu Qingfeng Sun Kai Zheng Xiubo Geng Pu Zhao Jiazhan Feng Chongyang Tao Daxin Jiang ALM 48 919 0 24 Apr 2023
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models Deyao Zhu Jun Chen Xiaoqian Shen Xiang Li Mohamed Elhoseiny VLM MLLM 75 1,922 0 20 Apr 2023