v1v2 (latest)

Unifying Vision-and-Language Tasks via Text Generation

4 February 2021

ArXiv (abs)PDF HTML Github (372★)

Papers citing "Unifying Vision-and-Language Tasks via Text Generation"

50 / 170 papers shown

Title
Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning Qian Jiang Changyou Chen Han Zhao Liqun Chen Q. Ping S. D. Tran Yi Xu Belinda Zeng Trishul Chilimbi 97 43 0 10 Mar 2023
MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering Jingjing Jiang Nanning Zheng MoE 112 6 0 02 Mar 2023
A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram Ming-Liang Zhang Fei Yin Cheng-Lin Liu AI4CE 167 45 0 22 Feb 2023
Retrieval-augmented Image Captioning R. Ramos Desmond Elliott Bruno Martins VLM 80 29 0 16 Feb 2023
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning Zhuolin Yang Ming-Yu Liu Zihan Liu V. Korthikanti Weili Nie ... Yuke Zhu Mohammad Shoeybi Bryan Catanzaro Chaowei Xiao Anima Anandkumar VLM RALM 108 40 0 09 Feb 2023
Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment Hao Liu Wilson Yan Pieter Abbeel 99 25 0 02 Feb 2023
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Xinsong Zhang Yan Zeng Jipeng Zhang Hang Li VLM AI4CE LRM 111 17 0 12 Jan 2023
SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph Yuxing Long Binyuan Hui Fulong Ye Yanyang Li Zhuoxin Han Caixia Yuan Yongbin Li Xiaojie Wang LLMAG 65 8 0 05 Jan 2023
GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods Da Yin Feng Gao Govind Thattai Michael F. Johnston Kai-Wei Chang VLM 88 15 0 05 Jan 2023
Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation Jianzong Wu Xiangtai Li Henghui Ding Xia Li Guangliang Cheng Yu Tong Chen Change Loy VLM 176 31 0 02 Jan 2023
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training Qinghao Ye Guohai Xu Ming Yan Haiyang Xu Qi Qian Ji Zhang Fei Huang VLM AI4TS 220 75 0 30 Dec 2022
Do DALL-E and Flamingo Understand Each Other? Hang Li Jindong Gu Rajat Koner Sahand Sharifzadeh Volker Tresp MLLM 79 12 0 23 Dec 2022
Generalized Decoding for Pixel, Image, and Language Xueyan Zou Zi-Yi Dou Jianwei Yang Zhe Gan Linjie Li ... Lu Yuan Nanyun Peng Lijuan Wang Yong Jae Lee Jianfeng Gao VLM MLLM ObjD 124 259 0 21 Dec 2022
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning Zhiyang Xu Ying Shen Lifu Huang MLLM 139 120 0 21 Dec 2022
A Survey of Deep Learning for Mathematical Reasoning Pan Lu Liang Qiu Wenhao Yu Sean Welleck Kai-Wei Chang ReLM LRM 133 150 0 20 Dec 2022
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering Fangyu Liu Francesco Piccinno Syrine Krichene Chenxi Pang Kenton Lee Mandar Joshi Yasemin Altun Nigel Collier Julian Martin Eisenschlos VLM LRM 57 102 0 19 Dec 2022
Objaverse: A Universe of Annotated 3D Objects Matt Deitke Dustin Schwenk Jordi Salvador Luca Weihs Oscar Michel Eli VanderBilt Ludwig Schmidt Kiana Ehsani Aniruddha Kembhavi Ali Farhadi 116 975 0 15 Dec 2022
Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization Yunlong Liang Fandong Meng Jinan Xu Jiaan Wang Jinan Xu Jie Zhou 101 22 0 15 Dec 2022
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning Hongkuan Zhang Saku Sugawara Akiko Aizawa Lei Zhou Ryohei Sasano Koichi Takeda VLM 80 4 0 14 Dec 2022
Learning Video Representations from Large Language Models Yue Zhao Ishan Misra Philipp Krahenbuhl Rohit Girdhar VLM AI4TS 118 177 0 08 Dec 2022
Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations Björn Plüster Jakob Ambsdorf Lukas Braach Jae Hee Lee S. Wermter 72 6 0 08 Dec 2022
UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression Jiaqi Chen Tong Li Jinghui Qin Pan Lu Liang Lin Chongyu Chen Xiaodan Liang AIMat LRM 109 105 0 06 Dec 2022
Unifying Vision, Text, and Layout for Universal Document Processing Zineng Tang Ziyi Yang Guoxin Wang Yuwei Fang Yang Liu Chenguang Zhu Michael Zeng Chao-Yue Zhang Joey Tianyi Zhou VLM 131 115 0 05 Dec 2022
Compound Tokens: Channel Fusion for Vision-Language Representation Learning Maxwell Mbabilla Aladago A. Piergiovanni 64 2 0 02 Dec 2022
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation Jiangyong Huang William Zhu Baoxiong Jia Zan Wang Xiaojian Ma Qing Li Siyuan Huang 123 5 0 28 Nov 2022
Understanding BLOOM: An empirical study on diverse NLP tasks Parag Dakle Sai Krishna Rallabandi Preethi Raghavan AI4CE 89 4 0 27 Nov 2022
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention Zineng Tang Jaemin Cho Jie Lei Joey Tianyi Zhou VLM 77 9 0 21 Nov 2022
A survey on knowledge-enhanced multimodal learning Maria Lymperaiou Giorgos Stamou 159 15 0 19 Nov 2022
CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge Linli Yao Wei Chen Qin Jin VLM 121 11 0 17 Nov 2022
PromptCap: Prompt-Guided Task-Aware Image Captioning Yushi Hu Hang Hua Zhengyuan Yang Weijia Shi Noah A. Smith Jiebo Luo 111 106 0 15 Nov 2022
A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation Shijia Huang Feng Li Hao Zhang Siyi Liu Lei Zhang Liwei Wang 64 5 0 15 Nov 2022
YORO -- Lightweight End to End Visual Grounding Chih-Hui Ho Srikar Appalaraju Bhavan A. Jasani R. Manmatha Nuno Vasconcelos ObjD 60 22 0 15 Nov 2022
Multi-VQG: Generating Engaging Questions for Multiple Images Min-Hsuan Yeh Vicent Chen Ting-Hao Haung Lun-Wei Ku CoGe 111 7 0 14 Nov 2022
Large-Scale Bidirectional Training for Zero-Shot Image Captioning Taehoon Kim Mark A Marsden Pyunghwan Ahn Sangyun Kim Sihaeng Lee Alessandra Sala S. Kim VLM 62 4 0 13 Nov 2022
Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detection Yanxin Long Jianhua Han Runhu Huang Xu Hang Yi Zhu Chunjing Xu Xiaodan Liang VLM ObjD 104 19 0 02 Nov 2022
Towards Unifying Reference Expression Generation and Comprehension Duo Zheng Tao Kong Ya Jing Jiaan Wang Xiaojie Wang ObjD 55 6 0 24 Oct 2022
Visual Spatial Description: Controlled Spatial-Oriented Image-to-Text Generation Yu Zhao Jianguo Wei Zhichao Lin Yueheng Sun Meishan Zhang Hao Fei 72 16 0 20 Oct 2022
Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training A. M. H. Tiong Junnan Li Boyang Albert Li Silvio Savarese Guosheng Lin MLLM 127 109 0 17 Oct 2022
Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training Wenliang Dai Zihan Liu Ziwei Ji Dan Su Pascale Fung MLLM VLM 86 67 0 14 Oct 2022
ImaginaryNet: Learning Object Detectors without Real Images and Annotations Minheng Ni Zitong Huang Kai-Hua Feng W. Zuo VLM 108 16 0 13 Oct 2022
OpenCQA: Open-ended Question Answering with Charts Shankar Kantharaj Do Xuan Long Rixie Tiffany Ko Leong J. Tan Enamul Hoque Shafiq Joty 83 53 0 12 Oct 2022
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks Gregor Geigle Chen Cecilia Liu Jonas Pfeiffer Iryna Gurevych VLM 50 1 0 12 Oct 2022
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation Wanrong Zhu An Yan Yujie Lu Wenda Xu Xinze Wang Miguel P. Eckstein William Yang Wang 126 36 0 07 Oct 2022
VIMA: General Robot Manipulation with Multimodal Prompts Yunfan Jiang Agrim Gupta Zichen Zhang Guanzhi Wang Yongqiang Dou Yanjun Chen Li Fei-Fei Anima Anandkumar Yuke Zhu Linxi Fan LM&Ro 115 355 0 06 Oct 2022
LAVIS: A Library for Language-Vision Intelligence Dongxu Li Junnan Li Hung Le Guangsen Wang Silvio Savarese Guosheng Lin VLM 192 56 0 15 Sep 2022
PaLI: A Jointly-Scaled Multilingual Language-Image Model Xi Chen Tianlin Li Soravit Changpinyo A. Piergiovanni Piotr Padlewski ... Andreas Steiner A. Angelova Xiaohua Zhai N. Houlsby Radu Soricut MLLM VLM 205 741 0 14 Sep 2022
Multimedia Generative Script Learning for Task Planning Qingyun Wang Manling Li Hou Pong Chan Lifu Huang Julia Hockenmaier Girish Chowdhary Heng Ji VGen 111 13 0 25 Aug 2022
Prompt Tuning for Generative Multimodal Pretrained Models Han Yang Junyang Lin An Yang Peng Wang Chang Zhou Hongxia Yang VLM LRM VPVLM 86 31 0 04 Aug 2022
Augmenting Vision Language Pretraining by Learning Codebook with Visual Semantics Xiaoyuan Guo Jiali Duan C.-C. Jay Kuo J. Gichoya Imon Banerjee VLM 46 1 0 31 Jul 2022
LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection Zhuo Chen Yufen Huang Jiaoyan Chen Yuxia Geng Yin Fang Jeff Z. Pan Ningyu Zhang Wen Zhang 95 38 0 26 Jul 2022