Quantifying Memorization and Retriever Performance in Retrieval-Augmented Vision-Language Models

20 February 2025

Papers citing "Quantifying Memorization and Retriever Performance in Retrieval-Augmented Vision-Language Models"

28 / 28 papers shown

Title
When to Retrieve: Teaching LLMs to Utilize Information Retrieval Effectively Tiziano Labruna Jon Ander Campos Gorka Azkune 40 11 0 30 Apr 2024
Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Marah Abdin Sam Ade Jacobs A. A. Awan J. Aneja Ahmed Hassan Awadallah ... Li Zhang Yi Zhang Yue Zhang Yunan Zhang Xiren Zhou LRM ALM 79 1,136 0 22 Apr 2024
Knowledge Conflicts for LLMs: A Survey Rongwu Xu Zehan Qi Zhijiang Guo Cunxiang Wang Hongru Wang Yue Zhang Wei Xu 229 104 0 13 Mar 2024
Unified Language Representation for Question Answering over Text, Tables, and Images Yu Bowen Cheng Fu Haiyang Yu Fei Huang Yongbin Li LMTD 59 21 0 29 Jun 2023
Why So Gullible? Enhancing the Robustness of Retrieval-Augmented Models against Counterfactual Noise Giwon Hong Jeonghwan Kim Junmo Kang Sung-Hyon Myaeng Joyce Jiyoung Whang RALM AAML 42 22 0 02 May 2023
In-Context Retrieval-Augmented Language Models Ori Ram Yoav Levine Itay Dalmedigos Dor Muhlgay Amnon Shashua Kevin Leyton-Brown Y. Shoham KELM RALM LRM 66 570 0 31 Jan 2023
SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images Ryota Tanaka Kyosuke Nishida Kosuke Nishida Taku Hasegawa Itsumi Saito Kuniko Saito 35 76 0 12 Jan 2023
DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answering Ella Neeman Roee Aharoni Or Honovich Leshem Choshen Idan Szpektor Omri Abend KELM CML 33 80 0 10 Nov 2022
Rich Knowledge Sources Bring Complex Knowledge Conflicts: Recalibrating Models to Reflect Conflicting Evidence Hung-Ting Chen Michael J.Q. Zhang Eunsol Choi RALM HILM 83 93 0 25 Oct 2022
VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment Shraman Pramanick Li Jing Sayan Nag Jiachen Zhu Hardik Shah Yann LeCun Ramalingam Chellappa 47 21 0 09 Oct 2022
MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Text Wenhu Chen Hexiang Hu Xi Chen Pat Verga William W. Cohen RALM 45 149 0 06 Oct 2022
GIT: A Generative Image-to-text Transformer for Vision and Language Jianfeng Wang Zhengyuan Yang Xiaowei Hu Linjie Li Kevin Qinghong Lin Zhe Gan Zicheng Liu Ce Liu Lijuan Wang VLM 94 537 0 27 May 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 438 4,283 0 28 Jan 2022
Entity-Based Knowledge Conflicts in Question Answering Shayne Longpre Kartik Perisetla Anthony Chen Nikhil Ramesh Chris DuBois Sameer Singh HILM 294 249 0 10 Sep 2021
LoRA: Low-Rank Adaptation of Large Language Models J. E. Hu Yelong Shen Phillip Wallis Zeyuan Allen-Zhu Yuanzhi Li Shean Wang Lu Wang Weizhu Chen OffRL AI4TS AI4CE ALM AIMat 230 10,099 0 17 Jun 2021
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows Ze Liu Yutong Lin Yue Cao Han Hu Yixuan Wei Zheng Zhang Stephen Lin B. Guo ViT 319 21,175 0 25 Mar 2021
Learning Transferable Visual Models From Natural Language Supervision Alec Radford Jong Wook Kim Chris Hallacy Aditya A. Ramesh Gabriel Goh ... Amanda Askell Pamela Mishkin Jack Clark Gretchen Krueger Ilya Sutskever CLIP VLM 673 28,659 0 26 Feb 2021
Graph Optimal Transport for Cross-Domain Alignment Liqun Chen Zhe Gan Yu Cheng Linjie Li Lawrence Carin Jingjing Liu OT 70 151 0 26 Jun 2020
Language Models are Few-Shot Learners Tom B. Brown Benjamin Mann Nick Ryder Melanie Subbiah Jared Kaplan ... Christopher Berner Sam McCandlish Alec Radford Ilya Sutskever Dario Amodei BDL 498 41,106 0 28 May 2020
Unified Vision-Language Pre-Training for Image Captioning and VQA Luowei Zhou Hamid Palangi Lei Zhang Houdong Hu Jason J. Corso Jianfeng Gao MLLM VLM 322 933 0 24 Sep 2019
RoBERTa: A Robustly Optimized BERT Pretraining Approach Yinhan Liu Myle Ott Naman Goyal Jingfei Du Mandar Joshi Danqi Chen Omer Levy M. Lewis Luke Zettlemoyer Veselin Stoyanov AIMat 406 24,160 0 26 Jul 2019
GOT: An Optimal Transport framework for Graph comparison Hermina Petric Maretic Mireille El Gheche Giovanni Chierchia P. Frossard OT 105 119 0 05 Jun 2019
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge Kenneth Marino Mohammad Rastegari Ali Farhadi Roozbeh Mottaghi 57 1,050 0 31 May 2019
A Corpus for Reasoning About Natural Language Grounded in Photographs Alane Suhr Stephanie Zhou Ally Zhang Iris Zhang Huajun Bai Yoav Artzi LRM 76 596 0 01 Nov 2018
HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering Zhilin Yang Peng Qi Saizheng Zhang Yoshua Bengio William W. Cohen Ruslan Salakhutdinov Christopher D. Manning RALM 115 2,577 0 25 Sep 2018
Attention Is All You Need Ashish Vaswani Noam M. Shazeer Niki Parmar Jakob Uszkoreit Llion Jones Aidan Gomez Lukasz Kaiser Illia Polosukhin 3DV 443 129,831 0 12 Jun 2017
Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering Yash Goyal Tejas Khot D. Summers-Stay Dhruv Batra Devi Parikh CoGe 289 3,187 0 02 Dec 2016
Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization Ramprasaath R. Selvaraju Michael Cogswell Abhishek Das Ramakrishna Vedantam Devi Parikh Dhruv Batra FAtt 211 19,796 0 07 Oct 2016