Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks

17 June 2022

Papers citing "Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks"

50 / 327 papers shown

Title
Tokenize Anything via Prompting Ting Pan Lulu Tang Xinlong Wang Shiguang Shan VLM 31 22 0 14 Dec 2023
Hallucination Augmented Contrastive Learning for Multimodal Large Language Model Chaoya Jiang Haiyang Xu Mengfan Dong Jiaxing Chen Wei Ye Mingshi Yan Qinghao Ye Ji Zhang Fei Huang Shikun Zhang VLM 20 51 0 12 Dec 2023
4M: Massively Multimodal Masked Modeling David Mizrahi Roman Bachmann Ouguzhan Fatih Kar Teresa Yeo Mingfei Gao Afshin Dehghan Amir Zamir MLLM 50 64 0 11 Dec 2023
Genixer: Empowering Multimodal Large Language Models as a Powerful Data Generator Henry Hengyuan Zhao Pan Zhou Mike Zheng Shou MLLM SyDa 38 7 0 11 Dec 2023
Visual Program Distillation: Distilling Tools and Programmatic Reasoning into Vision-Language Models Yushi Hu Otilia Stretcu Chun-Ta Lu Krishnamurthy Viswanathan Kenji Hata Enming Luo Ranjay Krishna Ariel Fuxman VLM LRM MLLM 52 29 0 05 Dec 2023
UPOCR: Towards Unified Pixel-Level OCR Interface Dezhi Peng Zhenhua Yang Jiaxin Zhang Chongyu Liu Yongxin Shi Kai Ding Fengjun Guo Lianwen Jin 34 10 0 05 Dec 2023
Lenna: Language Enhanced Reasoning Detection Assistant Fei Wei Xinyu Zhang Ailing Zhang Bo-Wen Zhang Xiangxiang Chu MLLM LRM 29 23 0 05 Dec 2023
GIVT: Generative Infinite-Vocabulary Transformers Michael Tschannen Cian Eastwood Fabian Mentzer 31 33 0 04 Dec 2023
Hulk: A Universal Knowledge Translator for Human-Centric Tasks Yizhou Wang YiXuan Wu Shixiang Tang Weizhen He Xun Guo ... Lei Bai Rui Zhao Jian Wu Tong He Wanli Ouyang VLM 46 14 0 04 Dec 2023
PixelLM: Pixel Reasoning with Large Multimodal Model Zhongwei Ren Zhicheng Huang Yunchao Wei Yao-Min Zhao Dongmei Fu Jiashi Feng Xiaojie Jin VLM MLLM LRM 33 82 0 04 Dec 2023
Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts Jialin Wu Xia Hu Yaqing Wang Bo Pang Radu Soricut MoE 24 14 0 01 Dec 2023
Manipulating the Label Space for In-Context Classification Haokun Chen Xu Yang Yuhang Huang Zihan Wu Jing Wang Xin Geng VLM 25 2 0 01 Dec 2023
InstructSeq: Unifying Vision Tasks with Instruction-conditioned Multi-modal Sequence Generation Rongyao Fang Shilin Yan Zhaoyang Huang Jingqiu Zhou Hao Tian Jifeng Dai Hongsheng Li MLLM 48 8 0 30 Nov 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Chenyu You Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 41 46 0 30 Nov 2023
Do text-free diffusion models learn discriminative visual representations? Soumik Mukhopadhyay M. Gwilliam Yosuke Yamaguchi Vatsal Agarwal Namitha Padmanabhan Archana Swaminathan Dinesh Manocha Abhinav Shrivastava DiffM 34 12 1 29 Nov 2023
ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model Fukun Yin Xin Chen C. Zhang Biao Jiang Zibo Zhao Jiayuan Fan Gang Yu Taihao Li Tao Chen 32 20 0 29 Nov 2023
E-ViLM: Efficient Video-Language Model via Masked Video Modeling with Semantic Vector-Quantized Tokenizer Jacob Zhiyuan Fang Skyler Zheng Vasu Sharma Robinson Piramuthu VLM 38 0 0 28 Nov 2023
Single-Model and Any-Modality for Video Object Tracking Zongwei Wu Jilai Zheng Xiangxuan Ren Florin-Alexandru Vasluianu Chao Ma D. Paudel Luc Van Gool Radu Timofte 61 35 0 27 Nov 2023
Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Models Xiaoyu Yang Lijian Xu Hao Sun Hongsheng Li Shaoting Zhang ObjD 33 6 0 21 Nov 2023
An Embodied Generalist Agent in 3D World Jiangyong Huang Silong Yong Xiaojian Ma Xiongkun Linghu Puhao Li Yan Wang Qing Li Song-Chun Zhu Baoxiong Jia Siyuan Huang LM&Ro 31 139 0 18 Nov 2023
DEED: Dynamic Early Exit on Decoder for Accelerating Encoder-Decoder Transformer Models Peng Tang Pengkai Zhu Tian Li Srikar Appalaraju Vijay Mahadevan R. Manmatha 57 7 0 15 Nov 2023
Vision-Language Instruction Tuning: A Review and Analysis Chen Li Yixiao Ge Dian Li Ying Shan VLM 39 12 0 14 Nov 2023
PerceptionGPT: Effectively Fusing Visual Perception into LLM Renjie Pi Lewei Yao Jiahui Gao Jipeng Zhang Tong Zhang MLLM 28 30 0 11 Nov 2023
Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Models Zhang Li Biao Yang Qiang Liu Zhiyin Ma Shuo Zhang Jingxu Yang Yabo Sun Yuliang Liu Xiang Bai MLLM 50 247 0 11 Nov 2023
Analyzing Modular Approaches for Visual Question Decomposition Apoorv Khandelwal Ellie Pavlick Chen Sun 45 4 0 10 Nov 2023
Florence-2: Advancing a Unified Representation for a Variety of Vision Tasks Bin Xiao Haiping Wu Weijian Xu Xiyang Dai Houdong Hu Yumao Lu Michael Zeng Ce Liu Lu Yuan VLM 45 143 0 10 Nov 2023
DAMEX: Dataset-aware Mixture-of-Experts for visual understanding of mixture-of-datasets Yash Jain Harkirat Singh Behl Z. Kira Vibhav Vineet 25 12 0 08 Nov 2023
TEAL: Tokenize and Embed ALL for Multi-modal Large Language Models Zhen Yang Yingxue Zhang Fandong Meng Jie Zhou VLM MLLM 47 3 0 08 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 126 379 0 07 Nov 2023
RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches Jiayuan Gu Sean Kirmani Paul Wohlhart Yao Lu Montse Gonzalez Arenas ... Hao Su Karol Hausman Chelsea Finn Q. Vuong Ted Xiao 36 63 0 03 Nov 2023
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editing Wei-Ge Chen Irina Spiridonova Jianwei Yang Jianfeng Gao Chun-yue Li MLLM VLM 13 34 0 01 Nov 2023
From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities Md Farhan Ishmam Md Sakib Hossain Shovon M. F. Mridha Nilanjan Dey 46 36 0 01 Nov 2023
Object-centric Video Representation for Long-term Action Anticipation Ce Zhang Changcheng Fu Shijie Wang Nakul Agarwal Kwonjoon Lee Chiho Choi Chen Sun 36 14 0 31 Oct 2023
Grounding Visual Illusions in Language: Do Vision-Language Models Perceive Illusions Like Humans? Yichi Zhang Jiayi Pan Yuchen Zhou Rui Pan Joyce Chai VLM 24 13 0 31 Oct 2023
Exploring Question Decomposition for Zero-Shot VQA Zaid Khan B. Vijaykumar S. Schulter Manmohan Chandraker Yun Fu ReLM 17 10 0 25 Oct 2023
Apollo: Zero-shot MultiModal Reasoning with Multiple Experts Daniela Ben-David Tzuf Paz-Argaman Reut Tsarfaty MoE 29 0 0 25 Oct 2023
CLIP meets Model Zoo Experts: Pseudo-Supervision for Visual Enhancement Mohammadreza Salehi Mehrdad Farajtabar Maxwell Horton Fartash Faghri Hadi Pouransari Raviteja Vemulapalli Oncel Tuzel Ali Farhadi Mohammad Rastegari Sachin Mehta CLIP VLM 48 1 0 21 Oct 2023
Visual Grounding Helps Learn Word Meanings in Low-Data Regimes Chengxu Zhuang Evelina Fedorenko Jacob Andreas 22 10 0 20 Oct 2023
Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning Jiachen Li Qiaozi Gao Michael Johnston Xiaofeng Gao Xuehai He Suhaila Shakiah Hangjie Shi R. Ghanadan William Y. Wang LM&Ro 27 12 0 14 Oct 2023
PolyTask: Learning Unified Policies through Behavior Distillation Siddhant Haldar Lerrel Pinto 28 7 0 12 Oct 2023
Ferret: Refer and Ground Anything Anywhere at Any Granularity Haoxuan You Haotian Zhang Zhe Gan Xianzhi Du Bowen Zhang Zirui Wang Liangliang Cao Shih-Fu Chang Yinfei Yang ObjD MLLM VLM 45 301 0 11 Oct 2023
Lightweight In-Context Tuning for Multimodal Unified Models Yixin Chen Shuai Zhang Boran Han Jiaya Jia 24 2 0 08 Oct 2023
VLATTACK: Multimodal Adversarial Attacks on Vision-Language Tasks via Pre-trained Models Ziyi Yin Muchao Ye Tianrong Zhang Tianyu Du Jinguo Zhu Han Liu Jinghui Chen Ting Wang Fenglong Ma AAML VLM CoGe 33 36 0 07 Oct 2023
Pink: Unveiling the Power of Referential Comprehension for Multi-modal LLMs Shiyu Xuan Qingpei Guo Ming Yang Shiliang Zhang MLLM ObjD 18 38 0 01 Oct 2023
InstructCV: Instruction-Tuned Text-to-Image Diffusion Models as Vision Generalists Yulu Gan Sungwoo Park Alexander Schubert Anthony Philippakis Ahmed Alaa VLM 38 22 0 30 Sep 2023
SCOB: Universal Text Understanding via Character-wise Supervised Contrastive Learning with Online Text Rendering for Bridging Domain Gap Daehee Kim Yoon Kim Donghyun Kim Yumin Lim Geewook Kim Taeho Kil 34 3 0 21 Sep 2023
DreamLLM: Synergistic Multimodal Comprehension and Creation Runpei Dong Chunrui Han Yuang Peng Zekun Qi Zheng Ge ... Hao-Ran Wei Xiangwen Kong Xiangyu Zhang Kaisheng Ma Li Yi MLLM 39 173 0 20 Sep 2023
RMT: Retentive Networks Meet Vision Transformers Qihang Fan Huaibo Huang Mingrui Chen Hongmin Liu Ran He ViT 43 75 0 20 Sep 2023
Frequency-Aware Masked Autoencoders for Multimodal Pretraining on Biosignals Ran Liu Ellen L. Zippi Hadi Pouransari Chris Sandino Jingping Nie Hanlin Goh Erdrin Azemi Ali Moin 39 12 0 12 Sep 2023
InstructDiffusion: A Generalist Modeling Interface for Vision Tasks Zigang Geng Binxin Yang Tiankai Hang Chen Li Shuyang Gu ... Jianmin Bao Zheng-Wei Zhang Han Hu Dongdong Chen Baining Guo DiffM VLM 53 93 0 07 Sep 2023