Language Models Can See: Plugging Visual Controls in Text Generation

5 May 2022

Lingpeng Kong

Papers citing "Language Models Can See: Plugging Visual Controls in Text Generation"

33 / 83 papers shown

Title
DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training Wei Li Linchao Zhu Longyin Wen Yi Yang VLM 50 86 0 06 Mar 2023
ConZIC: Controllable Zero-shot Image Captioning by Sampling-Based Polishing Zequn Zeng Hao Zhang Zhengjue Wang Ruiying Lu Dongsheng Wang Bo Chen BDL DiffM 24 33 0 04 Mar 2023
Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agents Wenlong Huang Fei Xia Dhruv Shah Danny Driess Andy Zeng ... Pete Florence Igor Mordatch Sergey Levine Karol Hausman Brian Ichter LM&Ro 32 43 0 01 Mar 2023
QR-CLIP: Introducing Explicit Open-World Knowledge for Location and Time Reasoning Weimin Shi Mingchen Zhuge D. Gao Zhong Zhou Ming-Ming Cheng Deng-Ping Fan LRM VLM 25 0 0 02 Feb 2023
DePlot: One-shot visual language reasoning by plot-to-table translation Fangyu Liu Julian Martin Eisenschlos Francesco Piccinno Syrine Krichene Chenxi Pang Kenton Lee Mandar Joshi Wenhu Chen Nigel Collier Yasemin Altun VLM ReLM LRM 30 89 0 20 Dec 2022
Plug-and-Play Recipe Generation with Content Planning Yinhong Liu Yixuan Su Ehsan Shareghi Nigel Collier 24 5 0 09 Dec 2022
Vision and Structured-Language Pretraining for Cross-Modal Food Retrieval Mustafa Shukor Nicolas Thome Matthieu Cord CLIP CoGe 37 8 0 08 Dec 2022
Momentum Decoding: Open-ended Text Generation As Graph Exploration Tian Lan Yixuan Su Shuhang Liu Heyan Huang Xian-Ling Mao 47 5 0 05 Dec 2022
An Empirical Study On Contrastive Search And Contrastive Decoding For Open-ended Text Generation Yixuan Su Jialu Xu 25 13 0 19 Nov 2022
I Can't Believe There's No Images! Learning Visual Tasks Using only Language Supervision Sophia Gu Christopher Clark Aniruddha Kembhavi VLM 32 24 0 17 Nov 2022
Zero-shot Image Captioning by Anchor-augmented Vision-Language Space Alignment Junyan Wang Yi Zhang Ming Yan Ji Zhang Jitao Sang VLM 36 9 0 14 Nov 2022
Text-Only Training for Image Captioning using Noise-Injected CLIP David Nukrai Ron Mokady Amir Globerson VLM CLIP 68 94 0 01 Nov 2022
Contrastive Search Is What You Need For Neural Text Generation Yixuan Su Nigel Collier 25 50 0 25 Oct 2022
Visualizing the Obvious: A Concreteness-based Ensemble Model for Noun Property Prediction Yue Yang Artemis Panagopoulou Marianna Apidianaki Mark Yatskar Chris Callison-Burch 29 2 0 24 Oct 2022
Describing Sets of Images with Textual-PCA Oded Hupert Idan Schwartz Lior Wolf CoGe 37 1 0 21 Oct 2022
Visualize Before You Write: Imagination-Guided Open-Ended Text Generation Wanrong Zhu An Yan Yujie Lu Wenda Xu Junfeng Fang Miguel P. Eckstein William Yang Wang 82 37 0 07 Oct 2022
What does a platypus look like? Generating customized prompts for zero-shot image classification Sarah M Pratt Ian Covert Rosanne Liu Ali Farhadi VLM 133 216 0 07 Sep 2022
Efficient Vision-Language Pretraining with Visual Concepts and Hierarchical Alignment Mustafa Shukor Guillaume Couairon Matthieu Cord VLM CLIP 24 27 0 29 Aug 2022
Zero-Shot Video Captioning with Evolving Pseudo-Tokens Yoad Tewel Yoav Shalev Roy Nadler Idan Schwartz Lior Wolf 37 27 0 22 Jul 2022
Scene-Aware Prompt for Multi-modal Dialogue Understanding and Generation Bin Li Yixuan Weng Ziyu Ma Bin Sun Shutao Li VLM 17 2 0 05 Jul 2022
Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning Yujia Xie Luowei Zhou Xiyang Dai Lu Yuan Nguyen Bach Ce Liu Michael Zeng VLM MLLM 37 28 0 03 Jun 2022
Language Models with Image Descriptors are Strong Few-Shot Video-Language Learners Zhenhailong Wang Manling Li Ruochen Xu Luowei Zhou Jie Lei ... Chenguang Zhu Derek Hoiem Shih-Fu Chang Joey Tianyi Zhou Heng Ji MLLM VLM 170 138 0 22 May 2022
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language Andy Zeng Maria Attarian Brian Ichter K. Choromanski Adrian S. Wong ... Michael S. Ryoo Vikas Sindhwani Johnny Lee Vincent Vanhoucke Peter R. Florence ReLM LRM 54 574 0 01 Apr 2022
TaCL: Improving BERT Pre-training with Token-aware Contrastive Learning Yixuan Su Fangyu Liu Zaiqiao Meng Tian Lan Lei Shu Ehsan Shareghi Nigel Collier 139 57 0 07 Nov 2021
Exploring Dense Retrieval for Dialogue Response Selection Tian Lan Deng Cai Yan Wang Yixuan Su Heyan Huang Xian-Ling Mao 120 16 0 13 Oct 2021
Multi-Task Pre-Training for Plug-and-Play Task-Oriented Dialogue System Yixuan Su Lei Shu Elman Mansimov Arshit Gupta Deng Cai Yi-An Lai Yi Zhang 150 192 0 29 Sep 2021
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 259 561 0 28 Sep 2021
How Much Can CLIP Benefit Vision-and-Language Tasks? Sheng Shen Liunian Harold Li Hao Tan Joey Tianyi Zhou Anna Rohrbach Kai-Wei Chang Z. Yao Kurt Keutzer CLIP VLM MLLM 202 406 0 13 Jul 2021
Zero-Shot Text-to-Image Generation Aditya A. Ramesh Mikhail Pavlov Gabriel Goh Scott Gray Chelsea Voss Alec Radford Mark Chen Ilya Sutskever VLM 255 4,816 0 24 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 340 3,726 0 11 Feb 2021
Unified Vision-Language Pre-Training for Image Captioning and VQA Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J. Corso Jianfeng Gao MLLM VLM 252 928 0 24 Sep 2019
A Style-Based Generator Architecture for Generative Adversarial Networks Tero Karras S. Laine Timo Aila 312 10,391 0 12 Dec 2018
Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning Jiasen Lu Caiming Xiong Devi Parikh R. Socher 85 1,442 0 06 Dec 2016