Unifying Vision-and-Language Tasks via Text Generation

4 February 2021

Papers citing "Unifying Vision-and-Language Tasks via Text Generation"

50 / 368 papers shown

Title
Contextually-rich human affect perception using multimodal scene information Digbalay Bose Rajat Hebbar Krishna Somandepalli Shrikanth Narayanan 27 3 0 13 Mar 2023
Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning Qian Jiang Changyou Chen Han Zhao Liqun Chen Q. Ping S. D. Tran Yi Xu Belinda Zeng Trishul Chilimbi 49 38 0 10 Mar 2023
Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training Lisai Zhang Qingcai Chen Zhijian Chen Yunpeng Han Zhonghua Li Bo Zhao VLM 25 1 0 09 Mar 2023
Prismer: A Vision-Language Model with Multi-Task Experts Shikun Liu Linxi Fan Edward Johns Zhiding Yu Chaowei Xiao Anima Anandkumar VLM MLLM 44 21 0 04 Mar 2023
MixPHM: Redundancy-Aware Parameter-Efficient Tuning for Low-Resource Visual Question Answering Jingjing Jiang Nanning Zheng MoE 40 6 0 02 Mar 2023
Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning Antoine Yang Arsha Nagrani Paul Hongsuck Seo Antoine Miech Jordi Pont-Tuset Ivan Laptev Josef Sivic Cordelia Schmid AI4TS VLM 39 221 0 27 Feb 2023
A Multi-Modal Neural Geometric Solver with Textual Clauses Parsed from Diagram Ming-Liang Zhang Fei Yin Cheng-Lin Liu AI4CE 53 40 0 22 Feb 2023
Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey Tianlin Li Guangyao Chen Guangwu Qian Pengcheng Gao Xiaoyong Wei Yaowei Wang Yonghong Tian Wen Gao AI4CE VLM 31 202 0 20 Feb 2023
Towards Unifying Medical Vision-and-Language Pre-training via Soft Prompts Zhihong Chen Shizhe Diao Benyou Wang Guanbin Li Xiang Wan MedIm 22 29 0 17 Feb 2023
Retrieval-augmented Image Captioning R. Ramos Desmond Elliott Bruno Martins VLM 32 29 0 16 Feb 2023
UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling Haoyu Lu Yuqi Huo Guoxing Yang Zhiwu Lu Wei Zhan Masayoshi Tomizuka Mingyu Ding 33 31 0 13 Feb 2023
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning Zhuolin Yang Ming-Yu Liu Zihan Liu V. Korthikanti Weili Nie ... Yuke Zhu M. Shoeybi Bryan Catanzaro Chaowei Xiao Anima Anandkumar VLM RALM 34 39 0 09 Feb 2023
Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classification Bohan Li Xiao Xu Xinghao Wang Yutai Hou Yunlong Feng Feng Wang Xuanliang Zhang Qingfu Zhu Wanxiang Che DiffM VLM 23 10 0 04 Feb 2023
Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment Hao Liu Wilson Yan Pieter Abbeel 28 24 0 02 Feb 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 278 4,244 0 30 Jan 2023
Towards a Unified Model for Generating Answers and Explanations in Visual Question Answering Chenxi Whitehouse Tillman Weyde Pranava Madhyastha LRM 44 3 0 25 Jan 2023
Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks Xinsong Zhang Yan Zeng Jipeng Zhang Hang Li VLM AI4CE LRM 16 17 0 12 Jan 2023
SPRING: Situated Conversation Agent Pretrained with Multimodal Questions from Incremental Layout Graph Yuxing Long Binyuan Hui Fulong Ye Yanyang Li Zhuoxin Han Caixia Yuan Yongbin Li Xiaojie Wang LLMAG 30 7 0 05 Jan 2023
GIVL: Improving Geographical Inclusivity of Vision-Language Models with Pre-Training Methods Da Yin Feng Gao Govind Thattai Michael F. Johnston Kai-Wei Chang VLM 32 15 0 05 Jan 2023
Betrayed by Captions: Joint Caption Grounding and Generation for Open Vocabulary Instance Segmentation Jianzong Wu Xiangtai Li Henghui Ding Xia Li Guangliang Cheng Yu Tong Chen Change Loy VLM 85 31 0 02 Jan 2023
HiTeA: Hierarchical Temporal-Aware Video-Language Pre-training Qinghao Ye Guohai Xu Ming Yan Haiyang Xu Qi Qian Ji Zhang Fei Huang VLM AI4TS 173 69 0 30 Dec 2022
Do DALL-E and Flamingo Understand Each Other? Hang Li Jindong Gu Rajat Koner Sahand Sharifzadeh Volker Tresp MLLM 21 12 0 23 Dec 2022
Generalized Decoding for Pixel, Image, and Language Xueyan Zou Zi-Yi Dou Jianwei Yang Zhe Gan Linjie Li ... Lu Yuan Nanyun Peng Lijuan Wang Yong Jae Lee Jianfeng Gao VLM MLLM ObjD 21 241 0 21 Dec 2022
From Images to Textual Prompts: Zero-shot VQA with Frozen Large Language Models Jiaxian Guo Junnan Li Dongxu Li A. M. H. Tiong Boyang Albert Li Dacheng Tao Steven C. H. Hoi VLM MLLM 32 107 0 21 Dec 2022
MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning Zhiyang Xu Ying Shen Lifu Huang MLLM 32 110 0 21 Dec 2022
A Survey of Deep Learning for Mathematical Reasoning Pan Lu Liang Qiu Wenhao Yu Sean Welleck Kai-Wei Chang ReLM LRM 37 137 0 20 Dec 2022
DePlot: One-shot visual language reasoning by plot-to-table translation Fangyu Liu Julian Martin Eisenschlos Francesco Piccinno Syrine Krichene Chenxi Pang Kenton Lee Mandar Joshi Wenhu Chen Nigel Collier Yasemin Altun VLM ReLM LRM 27 89 0 20 Dec 2022
MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering Fangyu Liu Francesco Piccinno Syrine Krichene Chenxi Pang Kenton Lee Mandar Joshi Yasemin Altun Nigel Collier Julian Martin Eisenschlos VLM LRM 19 89 0 19 Dec 2022
Objaverse: A Universe of Annotated 3D Objects Matt Deitke Dustin Schwenk Jordi Salvador Luca Weihs Oscar Michel Eli VanderBilt Ludwig Schmidt Kiana Ehsani Aniruddha Kembhavi Ali Farhadi 29 884 0 15 Dec 2022
Summary-Oriented Vision Modeling for Multimodal Abstractive Summarization Yunlong Liang Fandong Meng Jinan Xu Jiaan Wang Yufeng Chen Jie Zhou 33 19 0 15 Dec 2022
Cross-Modal Similarity-Based Curriculum Learning for Image Captioning Hongkuan Zhang Saku Sugawara Akiko Aizawa Lei Zhou Ryohei Sasano Koichi Takeda VLM 27 4 0 14 Dec 2022
Learning Video Representations from Large Language Models Yue Zhao Ishan Misra Philipp Krahenbuhl Rohit Girdhar VLM AI4TS 28 165 0 08 Dec 2022
Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations Björn Plüster Jakob Ambsdorf Lukas Braach Jae Hee Lee S. Wermter 25 6 0 08 Dec 2022
UniGeo: Unifying Geometry Logical Reasoning via Reformulating Mathematical Expression Jiaqi Chen Tong Li Jinghui Qin Pan Lu Liang Lin Chongyu Chen Xiaodan Liang AIMat LRM 47 89 0 06 Dec 2022
Unifying Vision, Text, and Layout for Universal Document Processing Zineng Tang Ziyi Yang Guoxin Wang Yuwei Fang Yang Liu Chenguang Zhu Michael Zeng Chao-Yue Zhang Joey Tianyi Zhou VLM 32 105 0 05 Dec 2022
Compound Tokens: Channel Fusion for Vision-Language Representation Learning Maxwell Mbabilla Aladago A. Piergiovanni 19 1 0 02 Dec 2022
SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding F. Bastani Piper Wolters Ritwik Gupta Joe Ferdinando Aniruddha Kembhavi 32 98 0 28 Nov 2022
Perceive, Ground, Reason, and Act: A Benchmark for General-purpose Visual Representation Jiangyong Huang William Zhu Baoxiong Jia Zan Wang Xiaojian Ma Qing Li Siyuan Huang 37 5 0 28 Nov 2022
Understanding BLOOM: An empirical study on diverse NLP tasks Parag Dakle Sai Krishna Rallabandi Preethi Raghavan AI4CE 39 3 0 27 Nov 2022
Knowledge Prompting for Few-shot Action Recognition Yuheng Shi Xinxiao Wu Hanxi Lin VLM 19 4 0 22 Nov 2022
Perceiver-VL: Efficient Vision-and-Language Modeling with Iterative Latent Attention Zineng Tang Jaemin Cho Jie Lei Joey Tianyi Zhou VLM 24 9 0 21 Nov 2022
A survey on knowledge-enhanced multimodal learning Maria Lymperaiou Giorgos Stamou 41 13 0 19 Nov 2022
I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision Sophia Gu Christopher Clark Aniruddha Kembhavi VLM 19 24 0 17 Nov 2022
CapEnrich: Enriching Caption Semantics for Web Images via Cross-modal Pre-trained Knowledge Linli Yao Wei Chen Qin Jin VLM 27 10 0 17 Nov 2022
PromptCap: Prompt-Guided Task-Aware Image Captioning Yushi Hu Hang Hua Zhengyuan Yang Weijia Shi Noah A. Smith Jiebo Luo 51 101 0 15 Nov 2022
A Unified Mutual Supervision Framework for Referring Expression Segmentation and Generation Shijia Huang Feng Li Hao Zhang Siyi Liu Lei Zhang Liwei Wang 30 5 0 15 Nov 2022
YORO -- Lightweight End to End Visual Grounding Chih-Hui Ho Srikar Appalaraju Bhavan A. Jasani R. Manmatha Nuno Vasconcelos ObjD 21 21 0 15 Nov 2022
Multi-VQG: Generating Engaging Questions for Multiple Images Min-Hsuan Yeh Vicent Chen Ting-Hao Haung Lun-Wei Ku CoGe 18 7 0 14 Nov 2022
Large-Scale Bidirectional Training for Zero-Shot Image Captioning Taehoon Kim Mark A Marsden Pyunghwan Ahn Sangyun Kim Sihaeng Lee Alessandra Sala S. Kim VLM 27 4 0 13 Nov 2022
ERNIE-UniX2: A Unified Cross-lingual Cross-modal Framework for Understanding and Generation Bin Shan Yaqian Han Weichong Yin Shuohuan Wang Yu Sun Hao Tian Hua-Hong Wu Haifeng Wang MLLM VLM 16 7 0 09 Nov 2022