OneLLM: One Framework to Align All Modalities with Language

10 January 2025

Papers citing "OneLLM: One Framework to Align All Modalities with Language"

50 / 168 papers shown

Title
Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation Shilin Yan Renrui Zhang Ziyu Guo Wenchao Chen Wei Zhang Hongyang Li Yu Qiao Hao Dong Zhongjiang He Peng Gao VOS 84 36 0 25 May 2023
ChatBridge: Bridging Modalities with Large Language Model as a Language Catalyst Zijia Zhao Longteng Guo Tongtian Yue Si-Qing Chen Shuai Shao Xinxin Zhu Zehuan Yuan Jing Liu MLLM 73 57 0 25 May 2023
PandaGPT: One Model To Instruction-Follow Them All Yixuan Su Tian Lan Huayang Li Jialu Xu Yan Wang Deng Cai MLLM 70 291 0 25 May 2023
VideoLLM: Modeling Video Sequence with Large Language Models Guo Chen Yin-Dong Zheng Jiahao Wang Jilan Xu Yifei Huang ... Yi Wang Yali Wang Yu Qiao Tong Lu Limin Wang MLLM 118 83 0 22 May 2023
Pengi: An Audio Language Model for Audio Tasks Soham Deshmukh Benjamin Elizalde Rita Singh Huaming Wang MLLM AuLLM 68 179 0 19 May 2023
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities Dong Zhang Shimin Li Xin Zhang Jun Zhan Pengyu Wang Yaqian Zhou Xipeng Qiu AuLLM MLLM 106 334 0 18 May 2023
Listen, Think, and Understand Yuan Gong Hongyin Luo Alexander H. Liu Leonid Karlinsky James R. Glass ELM MLLM LRM 90 156 0 18 May 2023
Self-Chained Image-Language Model for Video Localization and Question Answering Shoubin Yu Jaemin Cho Prateek Yadav Joey Tianyi Zhou 115 137 0 11 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 99 2,049 0 11 May 2023
VideoChat: Chat-Centric Video Understanding Kunchang Li Yinan He Yi Wang Yizhuo Li Wen Wang Ping Luo Yali Wang Limin Wang Yu Qiao MLLM 92 568 0 10 May 2023
ImageBind: One Embedding Space To Bind Them All Rohit Girdhar Alaaeldin El-Nouby Zhuang Liu Mannat Singh Kalyan Vasudev Alwala Armand Joulin Ishan Misra VLM 130 927 0 09 May 2023
X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages Feilong Chen Minglun Han Haozhi Zhao Qingyang Zhang Jing Shi Shuang Xu Bo Xu MLLM 61 124 0 07 May 2023
LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model Peng Gao Jiaming Han Renrui Zhang Ziyi Lin Shijie Geng ... Pan Lu Conghui He Xiangyu Yue Hongsheng Li Yu Qiao MLLM 88 578 0 28 Apr 2023
mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality Qinghao Ye Haiyang Xu Guohai Xu Jiabo Ye Ming Yan ... Junfeng Tian Qiang Qi Ji Zhang Feiyan Huang Jingren Zhou VLM MLLM 275 948 0 27 Apr 2023
MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models Deyao Zhu Jun Chen Xiaoqian Shen Xiang Li Mohamed Elhoseiny VLM MLLM 140 2,023 0 20 Apr 2023
Visual Instruction Tuning Haotian Liu Chunyuan Li Qingyang Wu Yong Jae Lee SyDa VLM MLLM 503 4,725 0 17 Apr 2023
VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Dataset Sihan Chen Xingjian He Longteng Guo Xinxin Zhu Weining Wang Jinhui Tang Jinhui Tang VLM 62 110 0 17 Apr 2023
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab Timothée Darcet Théo Moutakanni Huy Q. Vo Marc Szafraniec ... Hervé Jégou Julien Mairal Patrick Labatut Armand Joulin Piotr Bojanowski VLM CLIP SSL 304 3,383 0 14 Apr 2023
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research Xinhao Mei Chutong Meng Haohe Liu Qiuqiang Kong Tom Ko Chengqi Zhao Mark D. Plumbley Yuexian Zou Wenwu Wang 96 210 0 30 Mar 2023
LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention Renrui Zhang Jiaming Han Chris Liu Peng Gao Aojun Zhou Xiangfei Hu Shilin Yan Pan Lu Hongsheng Li Yu Qiao MLLM 141 775 0 28 Mar 2023
GPT-4 Technical Report OpenAI OpenAI OpenAI Josh Achiam Steven Adler Sandhini Agarwal Lama Ahmad ... Shengjia Zhao Tianhao Zheng Juntang Zhuang William Zhuk Barret Zoph LLMAG MLLM 1.4K 14,313 0 15 Mar 2023
LLaMA: Open and Efficient Foundation Language Models Hugo Touvron Thibaut Lavril Gautier Izacard Xavier Martinet Marie-Anne Lachaux ... Faisal Azhar Aurelien Rodriguez Armand Joulin Edouard Grave Guillaume Lample ALM PILM 1.4K 13,167 0 27 Feb 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 419 4,539 0 30 Jan 2023
Objaverse: A Universe of Annotated 3D Objects Matt Deitke Dustin Schwenk Jordi Salvador Luca Weihs Oscar Michel Eli VanderBilt Ludwig Schmidt Kiana Ehsani Aniruddha Kembhavi Ali Farhadi 88 948 0 15 Dec 2022
InternVideo: General Video Foundation Models via Generative and Discriminative Learning Yi Wang Kunchang Li Yizhuo Li Yinan He Bingkun Huang ... Junting Pan Jiashuo Yu Yali Wang Limin Wang Yu Qiao VLM VGen 113 326 0 06 Dec 2022
Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation Yusong Wu Kai Chen Tianyu Zhang Yuchen Hui Marianna Nezhurina Taylor Berg-Kirkpatrick Shlomo Dubnov CLIP 112 525 0 12 Nov 2022
LAION-5B: An open large-scale dataset for training next generation image-text models Christoph Schuhmann Romain Beaumont Richard Vencu Cade Gordon Ross Wightman ... Srivatsa Kundurthy Katherine Crowson Ludwig Schmidt R. Kaczmarczyk J. Jitsev VLM MLLM CLIP 150 3,444 0 16 Oct 2022
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark Ashwin Kalyan ELM ReLM LRM 261 1,240 0 20 Sep 2022
Video Dialog as Conversation about Objects Living in Space-Time H. Pham T. Le Vuong Le Tu Minh Phuong T. Tran 58 11 0 08 Jul 2022
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid 127 237 0 16 Jun 2022
A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge Dustin Schwenk Apoorv Khandelwal Christopher Clark Kenneth Marino Roozbeh Mottaghi 60 536 0 03 Jun 2022
GIT: A Generative Image-to-text Transformer for Vision and Language Jianfeng Wang Zhengyuan Yang Xiaowei Hu Linjie Li Kevin Qinghong Lin Zhe Gan Zicheng Liu Ce Liu Lijuan Wang VLM 125 549 0 27 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 361 3,532 0 29 Apr 2022
Clotho-AQA: A Crowdsourced Dataset for Audio Question Answering Samuel Lipping Parthasaarathy Sudarsanam Konstantinos Drossos Tuomas Virtanen 48 62 0 20 Apr 2022
Training Compute-Optimal Large Language Models Jordan Hoffmann Sebastian Borgeaud A. Mensch Elena Buchatskaya Trevor Cai ... Karen Simonyan Erich Elsen Jack W. Rae Oriol Vinyals Laurent Sifre AI4TS 191 1,944 0 29 Mar 2022
Learning to Answer Questions in Dynamic Audio-Visual Scenarios Guangyao Li Yake Wei Yapeng Tian Chenliang Xu Ji-Rong Wen Di Hu 98 149 0 26 Mar 2022
Omnivore: A Single Model for Many Visual Modalities Rohit Girdhar Mannat Singh Nikhil Ravi Laurens van der Maaten Armand Joulin Ishan Misra 255 235 0 20 Jan 2022
Video as Conditional Graph Hierarchy for Multi-Granular Question Answering Junbin Xiao Angela Yao Zhiyuan Liu Yicong Li Wei Ji Tat-Seng Chua 53 113 0 12 Dec 2021
PointCLIP: Point Cloud Understanding by CLIP Renrui Zhang Ziyu Guo Wei Zhang Kunchang Li Xupeng Miao Tengjiao Wang Yu Qiao Peng Gao Hongsheng Li VLM 3DPC 249 445 0 04 Dec 2021
Ego4D: Around the World in 3,000 Hours of Egocentric Video Kristen Grauman Andrew Westbury Eugene Byrne Zachary Chavis Antonino Furnari ... Mike Zheng Shou Antonio Torralba Lorenzo Torresani Mingfei Yan Jitendra Malik EgoV 380 1,086 0 13 Oct 2021
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans Ainaz Eftekhar Alexander Sax Roman Bachmann Jitendra Malik Amir Zamir MedIm 78 299 0 11 Oct 2021
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding Hu Xu Gargi Ghosh Po-Yao (Bernie) Huang Dmytro Okhonko Armen Aghajanyan Florian Metze Luke Zettlemoyer Florian Metze Luke Zettlemoyer Christoph Feichtenhofer CLIP VLM 309 578 0 28 Sep 2021
AudioCLIP: Extending CLIP to Image, Text and Audio A. Guzhov Federico Raue Jörn Hees Andreas Dengel CLIP VLM 106 366 0 24 Jun 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 393 10,301 0 17 Jun 2021
Image2Point: 3D Point-Cloud Understanding with 2D Image Pretrained Models Chenfeng Xu Shijia Yang Tomer Galanti Bichen Wu Xiangyu Yue Bohan Zhai Wei Zhan Peter Vajda Kurt Keutzer Masayoshi Tomizuka 3DPC 46 53 0 08 Jun 2021
NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions Junbin Xiao Xindi Shang Angela Yao Tat-Seng Chua 84 483 0 18 May 2021
CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval Huaishao Luo Lei Ji Ming Zhong Yang Chen Wen Lei Nan Duan Tianrui Li CLIP VLM 405 802 0 18 Apr 2021
Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval Max Bain Arsha Nagrani Gül Varol Andrew Zisserman VGen 136 1,172 0 01 Apr 2021
Vision Transformers for Dense Prediction René Ranftl Alexey Bochkovskiy V. Koltun ViT MDE 130 1,729 0 24 Mar 2021
Pretrained Transformers as Universal Computation Engines Kevin Lu Aditya Grover Pieter Abbeel Igor Mordatch 50 221 0 09 Mar 2021