v1v2v3 (latest)

Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation

12 February 2025

Mohammad Mahdi Abootorabi

Amirhosein Zobeiri

Mahdi Dehghani

Mohammadali Mohammadkhani

Papers citing "Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation"

50 / 187 papers shown

Title
SK-VQA: Synthetic Knowledge Generation at Scale for Training Context-Augmented Multimodal LLMs Xin Su Man Luo Kris W Pan Tien Pei Chou Vasudev Lal Phillip Howard 96 4 0 28 Jun 2024
RAVEN: Multitask Retrieval Augmented Vision-Language Learning Varun Nagaraj Rao Siddharth Choudhary Aditya Deshpande R. Satzoda Srikar Appalaraju RALM VLM 81 5 0 27 Jun 2024
ColPali: Efficient Document Retrieval with Vision Language Models Manuel Faysse Hugues Sibille Tony Wu Bilel Omrani Gautier Viaud C´eline Hudelot Pierre Colombo VLM 255 29 0 27 Jun 2024
DrVideo: Document Retrieval Based Long Video Understanding Ziyu Ma Chenhui Gou Hengcan Shi Bin Sun Shutao Li Hamid Rezatofighi Jianfei Cai VLM 71 15 0 18 Jun 2024
PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers Myeonghwa Lee Seonho An Min-Soo Kim 3DV RALM 64 22 0 18 Jun 2024
MINT-1T: Scaling Open-Source Multimodal Data by 10x: A Multimodal Dataset with One Trillion Tokens Anas Awadalla Le Xue Oscar Lo Manli Shu Hannah Lee ... Silvio Savarese Caiming Xiong Ran Xu Yejin Choi Ludwig Schmidt 89 28 0 17 Jun 2024
OmniCorpus: A Unified Multimodal Corpus of 10 Billion-Level Images Interleaved with Text Qingyun Li Zhe Chen Weiyun Wang Wenhai Wang Shenglong Ye ... Dahua Lin Yu Qiao Botian Shi Conghui He Jifeng Dai VLM OffRL 86 27 0 12 Jun 2024
RU-AI: A Large Multimodal Dataset for Machine Generated Content Detection Liting Huang Zhihao Zhang Yiran Zhang Xiyue Zhou Shoujin Wang NoLa 70 4 0 07 Jun 2024
VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval Yueze Wang Zheng Liu Shitao Xiao Bo Zhao Yongping Xiong 85 28 0 06 Jun 2024
XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags Faisal Tareque Shohan Mir Tafseer Nayeem Samsul Islam Abu Ubaida Akash Shafiq Joty 71 4 0 06 Jun 2024
Retrieval Augmented Generation in Prompt-based Text-to-Speech Synthesis with Context-Aware Contrastive Language-Audio Pretraining Jinlong Xue Yayue Deng Yingming Gao Ya Li RALM VLM 124 6 0 06 Jun 2024
Don't Forget to Connect! Improving RAG with Graph-based Reranking Jialin Dong Bahare Fatemi Bryan Perozzi Lin F. Yang Anton Tsitsulin 109 29 0 28 May 2024
Visual Delta Generator with Large Multi-modal Models for Semi-supervised Composed Image Retrieval Young Kyun Jang Donghyun Kim Zihang Meng Dat Huynh Ser-Nam Lim 75 12 0 23 Apr 2024
Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs Davide Caffagni Federico Cocchi Nicholas Moratelli Sara Sarto Marcella Cornia Lorenzo Baraldi Rita Cucchiara KELM 96 46 0 23 Apr 2024
How Does the Textual Information Affect the Retrieval of Multimodal In-Context Learning? Yang Luo Zangwei Zheng Zirui Zhu Yang You 51 5 0 19 Apr 2024
Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval Minkuk Kim Hyeon Bae Kim Jinyoung Moon Jinwoo Choi Seong Tae Kim 48 22 0 11 Apr 2024
MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding Bo He Hengduo Li Young Kyun Jang Menglin Jia Xuefei Cao Ashish Shah Abhinav Shrivastava Ser-Nam Lim MLLM 123 100 0 08 Apr 2024
Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrieval Jiamian Wang Guohao Sun Pichao Wang Dongfang Liu S. Dianat Majid Rabbani Raghuveer M. Rao Zhiqiang Tao VGen 98 23 0 26 Mar 2024
Visually Guided Generative Text-Layout Pre-training for Document Intelligence Zhiming Mao Haoli Bai Lu Hou Jiansheng Wei Xin Jiang Qun Liu Kam-Fai Wong 62 10 0 25 Mar 2024
mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding Anwen Hu Haiyang Xu Jiabo Ye Mingshi Yan Liang Zhang ... Chen Li Ji Zhang Qin Jin Fei Huang Jingren Zhou VLM 94 123 0 19 Mar 2024
Improving Medical Multi-modal Contrastive Learning with Expert Annotations Yogesh Kumar Pekka Marttinen MedIm VLM 77 11 0 15 Mar 2024
RAFT: Adapting Language Model to Domain Specific RAG Tianjun Zhang Shishir G. Patil Naman Jain Sheng Shen Matei A. Zaharia Ion Stoica Joseph E. Gonzalez RALM 92 207 0 15 Mar 2024
RA-ISF: Learning to Answer and Understand from Retrieval Augmentation via Iterative Self-Feedback Yanming Liu Xinyue Peng Xuhong Zhang Weihao Liu Jianwei Yin Jiannan Cao Tianyu Du RALM 55 43 0 11 Mar 2024
Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control Thong Nguyen Mariya Hendriksen Andrew Yates Maarten de Rijke 76 6 0 27 Feb 2024
MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning Wanqing Cui Keping Bi Jiafeng Guo Xueqi Cheng SyDa ReLM RALM LRM 81 9 0 21 Feb 2024
PreFLMR: Scaling Up Fine-Grained Late-Interaction Multi-modal Retrievers Weizhe Lin Jingbiao Mei Jinghong Chen Bill Byrne VLM AI4Ed 107 20 0 13 Feb 2024
BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation Jianlv Chen Shitao Xiao Peitian Zhang Kun Luo Defu Lian Zheng Liu 645 433 0 05 Feb 2024
M2-RAAP: A Multi-Modal Recipe for Advancing Adaptation-based Pre-training towards Effective and Efficient Zero-shot Video-text Retrieval Xingning Dong Zipeng Feng Chunluan Zhou Xuzheng Yu Ming Yang Qingpei Guo VLM 68 3 0 31 Jan 2024
InFoBench: Evaluating Instruction Following Ability in Large Language Models Yiwei Qin Kaiqiang Song Yebowen Hu Wenlin Yao Sangwoo Cho Xiaoyang Wang Xuansheng Wu Fei Liu Pengfei Liu Dong Yu ELM 72 50 0 07 Jan 2024
DocLLM: A layout-aware generative language model for multimodal document understanding Dongsheng Wang Natraj Raman Mathieu Sibue Zhiqiang Ma Petr Babkin Simerjot Kaur Yulong Pei Armineh Nourbakhsh Xiaomo Liu VLM 80 60 0 31 Dec 2023
Retrieval-Augmented Generation for Large Language Models: A Survey Yunfan Gao Yun Xiong Xinyu Gao Kangxiang Jia Jinliu Pan Yuxi Bi Yi Dai Jiawei Sun Meng Wang Haofen Wang 3DV RALM 203 1,814 1 18 Dec 2023
UFineBench: Towards Text-based Person Retrieval with Ultra-fine Granularity Jia-li Zuo Hanyu Zhou Ying Nie Feng Zhang Tianyu Guo Nong Sang Yunhe Wang Changxin Gao 91 21 0 06 Dec 2023
C3Net: Compound Conditioned ControlNet for Multimodal Content Generation Juntao Zhang Yuehuai Liu Yu-Wing Tai Chi-Keung Tang DiffM 70 5 0 29 Nov 2023
UniIR: Training and Benchmarking Universal Multimodal Information Retrievers Cong Wei Yang Chen Haonan Chen Hexiang Hu Ge Zhang Jie Fu Alan Ritter Wenhu Chen 68 70 0 28 Nov 2023
ShareGPT4V: Improving Large Multi-Modal Models with Better Captions Lin Chen Jinsong Li Xiao-wen Dong Pan Zhang Conghui He Jiaqi Wang Feng Zhao Dahua Lin MLLM VLM 195 680 0 21 Nov 2023
Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection Akari Asai Zeqiu Wu Yizhong Wang Avirup Sil Hannaneh Hajishirzi RALM 265 776 0 17 Oct 2023
NExT-GPT: Any-to-Any Multimodal LLM Shengqiong Wu Hao Fei Leigang Qu Wei Ji Tat-Seng Chua MLLM 93 504 0 11 Sep 2023
Towards General Text Embeddings with Multi-stage Contrastive Learning Zehan Li Xin Zhang Yanzhao Zhang Dingkun Long Pengjun Xie Meishan Zhang 137 414 0 07 Aug 2023
Llama 2: Open Foundation and Fine-Tuned Chat Models Hugo Touvron Louis Martin Kevin R. Stone Peter Albert Amjad Almahairi ... Sharan Narang Aurelien Rodriguez Robert Stojnic Sergey Edunov Thomas Scialom AI4MH ALM 364 12,044 0 18 Jul 2023
MMBench: Is Your Multi-modal Model an All-around Player? Yuanzhan Liu Haodong Duan Yuanhan Zhang Yue Liu Songyang Zhang ... Jiaqi Wang Conghui He Ziwei Liu Kai-xiang Chen Dahua Lin 114 1,054 0 12 Jul 2023
Quilt-1M: One Million Image-Text Pairs for Histopathology Wisdom O. Ikezogwo M. S. Seyfioglu Fatemeh Ghezloo Dylan Stefan Chan Geva Fatwir Sheikh Mohammed Pavan Kumar Anand Ranjay Krishna Linda G. Shapiro CLIP VLM 286 125 0 20 Jun 2023
Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks Haiyang Xu Qinghao Ye Xuan-Wei Wu Mingshi Yan Yuan Miao ... Qingfang Qian Maofei Que Ji Zhang Xiaoyan Zeng Feiyan Huang VLM MLLM 83 25 0 07 Jun 2023
ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities Peng Wang Shijie Wang Junyang Lin Shuai Bai Xiaohuan Zhou Jingren Zhou Xinggang Wang Chang Zhou VLM MLLM ObjD 110 122 0 18 May 2023
PaLM 2 Technical Report Rohan Anil Andrew M. Dai Orhan Firat Melvin Johnson Dmitry Lepikhin ... Ce Zheng Wei Zhou Denny Zhou Slav Petrov Yonghui Wu ReLM LRM 208 1,204 0 17 May 2023
InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning Wenliang Dai Junnan Li Dongxu Li A. M. H. Tiong Junqi Zhao Weisheng Wang Boyang Albert Li Pascale Fung Steven C. H. Hoi MLLM VLM 134 2,095 0 11 May 2023
Otter: A Multi-Modal Model with In-Context Instruction Tuning Yue Liu Yuanhan Zhang Liangyu Chen Jinghao Wang Jingkang Yang Ziwei Liu MLLM 73 520 0 05 May 2023
DINOv2: Learning Robust Visual Features without Supervision Maxime Oquab Timothée Darcet Théo Moutakanni Huy Q. Vo Marc Szafraniec ... Hervé Jégou Julien Mairal Patrick Labatut Armand Joulin Piotr Bojanowski VLM CLIP SSL 364 3,489 0 14 Apr 2023
WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research Xinhao Mei Chutong Meng Haohe Liu Qiuqiang Kong Tom Ko Chengqi Zhao Mark D. Plumbley Yuexian Zou Wenwu Wang 124 218 0 30 Mar 2023
Zero-Shot Composed Image Retrieval with Textual Inversion Alberto Baldrati Lorenzo Agnolucci Marco Bertini A. Bimbo 79 114 0 27 Mar 2023
Retrieving Multimodal Information for Augmented Generation: A Survey Ruochen Zhao Hailin Chen Weishi Wang Fangkai Jiao Do Xuan Long ... Bosheng Ding Xiaobao Guo Minzhi Li Xingxuan Li Shafiq Joty 98 87 0 20 Mar 2023