ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

18 May 2023

Peng Wang

Shijie Wang

Junyang Lin

Shuai Bai

Xiaohuan Zhou

Jingren Zhou

Chang Zhou

Papers citing "ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities"

47 / 97 papers shown

Title
SC-Tune: Unleashing Self-Consistent Referential Comprehension in Large Vision Language Models Tongtian Yue Jie Cheng Longteng Guo Xingyuan Dai Zijia Zhao Xingjian He Gang Xiong Yisheng Lv Jing Liu 43 9 0 20 Mar 2024
Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval Qian Wang Jia-Chen Gu Zhen-Hua Ling 35 2 0 15 Mar 2024
VEglue: Testing Visual Entailment Systems via Object-Aligned Joint Erasing Zhiyuan Chang Mingyang Li Junjie Wang Cheng Li Qing Wang 22 0 0 05 Mar 2024
StochCA: A Novel Approach for Exploiting Pretrained Models with Cross-Attention SeungWon Seo Suho Lee Sangheum Hwang 32 0 0 25 Feb 2024
Latency-aware Road Anomaly Segmentation in Videos: A Photorealistic Dataset and New Metrics Beiwen Tian Huan-ang Gao Leiyao Cui Yupeng Zheng Lan Luo Baofeng Wang Rong Zhi Guyue Zhou Hao Zhao 26 4 0 10 Jan 2024
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action Jiasen Lu Christopher Clark Sangho Lee Zichen Zhang Savya Khosla Ryan Marten Derek Hoiem Aniruddha Kembhavi VLM MLLM 40 144 0 28 Dec 2023
InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks Zhe Chen Jiannan Wu Wenhai Wang Weijie Su Guo Chen ... Bin Li Ping Luo Tong Lu Yu Qiao Jifeng Dai VLM MLLM 176 924 0 21 Dec 2023
Data-Efficient Multimodal Fusion on a Single GPU Noël Vouitsis Zhaoyan Liu S. Gorti Valentin Villecroze Jesse C. Cresswell Guangwei Yu G. Loaiza-Ganem M. Volkovs 51 3 0 15 Dec 2023
AI-SAM: Automatic and Interactive Segment Anything Model Yimu Pan Sitao Zhang Alison D. Gernand Jeffery A. Goldstein J. Z. Wang VLM 32 4 0 05 Dec 2023
Rejuvenating image-GPT as Strong Visual Representation Learners Sucheng Ren Zeyu Wang Hongru Zhu Junfei Xiao Alan L. Yuille Cihang Xie VLM 57 7 0 04 Dec 2023
X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoning Artemis Panagopoulou Le Xue Ning Yu Junnan Li Dongxu Li Shafiq R. Joty Ran Xu Silvio Savarese Caiming Xiong Juan Carlos Niebles VLM MLLM 41 45 0 30 Nov 2023
ViT-Lens: Towards Omni-modal Representations Weixian Lei Yixiao Ge Kun Yi Jianfeng Zhang Difei Gao Dylan Sun Yuying Ge Ying Shan Mike Zheng Shou 21 18 0 27 Nov 2023
Mirasol3B: A Multimodal Autoregressive model for time-aligned and contextual modalities A. Piergiovanni Isaac Noble Dahun Kim Michael S. Ryoo Victor Gomes A. Angelova 36 19 0 09 Nov 2023
CogVLM: Visual Expert for Pretrained Language Models Weihan Wang Qingsong Lv Wenmeng Yu Wenyi Hong Ji Qi ... Bin Xu Juanzi Li Yuxiao Dong Ming Ding Jie Tang VLM MLLM 29 446 0 06 Nov 2023
From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunities Md Farhan Ishmam Md Sakib Hossain Shovon M. F. Mridha Nilanjan Dey 40 36 0 01 Nov 2023
Dynamic Convolutional Neural Networks as Efficient Pre-trained Audio Models Florian Schmid Khaled Koutini Gerhard Widmer 18 11 0 24 Oct 2023
CLARA: Multilingual Contrastive Learning for Audio Representation Acquisition K. A. Noriy Xiaosong Yang Marcin Budka Jian Jun Zhang VLM 26 3 0 18 Oct 2023
ViT-A: Legged Robot Path Planning using Vision Transformer A Jianwei Liu Shirui Lyu Denis Hadjivelichkov Valerio Modugno Dimitrios Kanoulas 40 8 0 11 Oct 2023
Text-image Alignment for Diffusion-based Perception Neehar Kondapaneni Markus Marks Manuel Knott Rogério Guimarães Pietro Perona VLM DiffM 24 32 0 29 Sep 2023
A Survey on Image-text Multimodal Models Ruifeng Guo Jingxuan Wei Linzhuang Sun Khai Le-Duc Guiyong Chang Dawei Liu Sibo Zhang Zhengbing Yao Mingjun Xu Liping Bu VLM 31 5 0 23 Sep 2023
Killing two birds with one stone: Can an audio captioning system also be used for audio-text retrieval? Etienne Labbé Thomas Pellegrini J. Pinquier 15 5 0 29 Aug 2023
Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond Jinze Bai Shuai Bai Shusheng Yang Shijie Wang Sinan Tan Peng Wang Junyang Lin Chang Zhou Jingren Zhou MLLM VLM ObjD 50 796 0 24 Aug 2023
Audio Generation with Multiple Conditional Diffusion Model Zhifang Guo Jianguo Mao Ruijie Tao Long Yan Kazushige Ouchi Hong Liu Xiangdong Wang DiffM 24 11 0 23 Aug 2023
ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights Weixian Lei Yixiao Ge Jianfeng Zhang Dylan Sun Kun Yi Ying Shan Mike Zheng Shou 33 1 0 20 Aug 2023
Applicability of scaling laws to vision encoding models Takuya Matsuyama K. Sasaki Shinji Nishimoto MedIm 21 4 0 01 Aug 2023
Vision Language Transformers: A Survey Clayton Fields C. Kennington VLM 28 5 0 06 Jul 2023
Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic Ke Chen Zhao Zhang Weili Zeng Richong Zhang Feng Zhu Rui Zhao ObjD 42 598 0 27 Jun 2023
Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding Hang Zhang Xin Li Lidong Bing MLLM 53 956 0 05 Jun 2023
VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Dataset Sihan Chen Handong Li Qunbo Wang Zijia Zhao Ming-Ting Sun Xinxin Zhu Jiaheng Liu 32 97 0 29 May 2023
An Empirical Study of Multimodal Model Merging Yi-Lin Sung Linjie Li Kevin Qinghong Lin Zhe Gan Joey Tianyi Zhou Lijuan Wang MoMe 20 40 0 28 Apr 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 208 900 0 27 Apr 2023
SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training Zi-Hua Zhang Long Zhou Junyi Ao Shujie Liu Lirong Dai Jinyu Li Furu Wei 61 57 0 07 Oct 2022
Audio Retrieval with WavText5K and CLAP Training Soham Deshmukh Benjamin Elizalde Huaming Wang 3DV CLIP 124 50 0 28 Sep 2022
Exploring Target Representations for Masked Autoencoders Xingbin Liu Jinghao Zhou Tao Kong Xianming Lin Rongrong Ji 100 50 0 08 Sep 2022
Contrastive Learning Rivals Masked Image Modeling in Fine-tuning via Feature Distillation Yixuan Wei Han Hu Zhenda Xie Zheng-Wei Zhang Yue Cao Jianmin Bao Dong Chen B. Guo CLIP 88 124 0 27 May 2022
HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection Ke Chen Xingjian Du Bilei Zhu Zejun Ma Taylor Berg-Kirkpatrick Shlomo Dubnov ViT 121 264 0 02 Feb 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,137 0 28 Jan 2022
Masked Autoencoders Are Scalable Vision Learners Kaiming He Xinlei Chen Saining Xie Yanghao Li Piotr Dollár Ross B. Girshick ViT TPM 305 7,443 0 11 Nov 2021
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing Junyi Ao Rui Wang Long Zhou Chengyi Wang Shuo Ren ... Yu Zhang Zhihua Wei Yao Qian Jinyu Li Furu Wei 118 193 0 14 Oct 2021
Emerging Properties in Self-Supervised Vision Transformers Mathilde Caron Hugo Touvron Ishan Misra Hervé Jégou Julien Mairal Piotr Bojanowski Armand Joulin 332 5,785 0 29 Apr 2021
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text Hassan Akbari Liangzhe Yuan Rui Qian Wei-Hong Chuang Shih-Fu Chang Huayu Chen Boqing Gong ViT 251 577 0 22 Apr 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 313 3,708 0 11 Feb 2021
Unifying Vision-and-Language Tasks via Text Generation Jaemin Cho Jie Lei Hao Tan Joey Tianyi Zhou MLLM 262 525 0 04 Feb 2021
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation Golnaz Ghiasi Huayu Chen A. Srinivas Rui Qian Nayeon Lee E. D. Cubuk Quoc V. Le Barret Zoph ISeg 252 969 0 13 Dec 2020
Improved Baselines with Momentum Contrastive Learning Xinlei Chen Haoqi Fan Ross B. Girshick Kaiming He SSL 270 3,375 0 09 Mar 2020
Semantic Understanding of Scenes through the ADE20K Dataset Bolei Zhou Hang Zhao Xavier Puig Tete Xiao Sanja Fidler Adela Barriuso Antonio Torralba SSeg 253 1,828 0 18 Aug 2016
ImageNet Large Scale Visual Recognition Challenge Olga Russakovsky Jia Deng Hao Su J. Krause S. Satheesh ... A. Karpathy A. Khosla Michael S. Bernstein Alexander C. Berg Li Fei-Fei VLM ObjD 296 39,198 0 01 Sep 2014