MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning

9 December 2021

Papers citing "MAGMA -- Multimodal Augmentation of Generative Models through Adapter-based Finetuning"

37 / 37 papers shown

Title
CTRL-O: Language-Controllable Object-Centric Visual Representation Learning Aniket Didolkar Andrii Zadaianchuk Rabiul Awal Maximilian Seitzer E. Gavves Aishwarya Agrawal OCL VLM 89 2 0 27 Mar 2025
A Concept-Based Explainability Framework for Large Multimodal Models Jayneel Parekh Pegah Khayatan Mustafa Shukor A. Newson Matthieu Cord 40 16 0 12 Jun 2024
Topological Perspectives on Optimal Multimodal Embedding Spaces Abdul Aziz Abdul Rahim BDL 45 0 0 29 May 2024
Do Vision & Language Decoders use Images and Text equally? How Self-consistent are their Explanations? Letitia Parcalabescu Anette Frank MLLM CoGe VLM 84 3 0 29 Apr 2024
Jack of All Tasks, Master of Many: Designing General-purpose Coarse-to-Fine Vision-Language Model Shraman Pramanick Guangxing Han Rui Hou Sayan Nag Ser-Nam Lim Nicolas Ballas Qifan Wang Rama Chellappa Amjad Almahairi VLM MLLM 48 29 0 19 Dec 2023
Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Prompts Bipin Rajendran Bashir M. Al-Hashimi MLLM VLM 32 2 0 27 Sep 2023
A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models Noriyuki Kojima Hadar Averbuch-Elor Yoav Artzi 31 2 0 06 Sep 2023
Bridging the Gap: Exploring the Capabilities of Bridge-Architectures for Complex Visual Reasoning Tasks Kousik Rajesh Mrigank Raman M. A. Karim Pranit Chawla VLM 25 2 0 31 Jul 2023
Linear Alignment of Vision-language Models for Image Captioning Fabian Paischer M. Hofmarcher Sepp Hochreiter Thomas Adler CLIP VLM 50 0 0 10 Jul 2023
Investigating Prompting Techniques for Zero- and Few-Shot Visual Question Answering Rabiul Awal Le Zhang Aishwarya Agrawal LRM 46 12 0 16 Jun 2023
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models Shuo Chen Jindong Gu Zhen Han Yunpu Ma Philip Torr Volker Tresp VPVLM VLM 42 17 0 03 Jun 2023
MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation Marco Bellagente Manuel Brack H. Teufel Felix Friedrich Bjorn Deiseroth ... Koen Oostermeijer Andres Felipe Cruz Salinas P. Schramowski Kristian Kersting Samuel Weinbach 45 16 0 24 May 2023
Making the Most of What You Have: Adapting Pre-trained Visual Language Models in the Low-data Regime Chuhan Zhang Antoine Miech Jiajun Shen Jean-Baptiste Alayrac Pauline Luc VLM VPVLM 47 2 0 03 May 2023
Towards ethical multimodal systems Alexis Roger Esma Aïmeur Irina Rish 40 3 0 26 Apr 2023
Towards Robust Prompts on Vision-Language Models Jindong Gu Ahmad Beirami Xuezhi Wang Alex Beutel Philip Torr Yao Qin VLM VPVLM 38 8 0 17 Apr 2023
AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASR Paul Hongsuck Seo Arsha Nagrani Cordelia Schmid 29 15 0 29 Mar 2023
Prismer: A Vision-Language Model with Multi-Task Experts Shikun Liu Linxi Fan Edward Johns Zhiding Yu Chaowei Xiao Anima Anandkumar VLM MLLM 44 21 0 04 Mar 2023
Meta Learning to Bridge Vision and Language Models for Multimodal Few-Shot Learning Ivona Najdenkoska Xiantong Zhen M. Worring VLM 21 18 0 28 Feb 2023
Re-ViLM: Retrieval-Augmented Visual Language Model for Zero and Few-Shot Image Captioning Zhuolin Yang Ming-Yu Liu Zihan Liu V. Korthikanti Weili Nie ... Yuke Zhu M. Shoeybi Bryan Catanzaro Chaowei Xiao Anima Anandkumar VLM RALM 34 39 0 09 Feb 2023
Aligning MAGMA by Few-Shot Learning and Finetuning Jean-Charles Layoun Alexis Roger Irina Rish VLM 24 2 0 18 Oct 2022
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks Gregor Geigle Chen Cecilia Liu Jonas Pfeiffer Iryna Gurevych VLM 28 1 0 12 Oct 2022
Does CLIP Know My Face? Dominik Hintersdorf Lukas Struppek Manuel Brack Felix Friedrich P. Schramowski Kristian Kersting VLM 21 9 0 15 Sep 2022
Zero-Shot Video Question Answering via Frozen Bidirectional Language Models Antoine Yang Antoine Miech Josef Sivic Ivan Laptev Cordelia Schmid 41 228 0 16 Jun 2022
On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization Shruti Palaskar Akshita Bhagia Yonatan Bisk Florian Metze A. Black Ana Marasović 31 4 0 24 May 2022
Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac Jeff Donahue Pauline Luc Antoine Miech Iain Barr ... Mikolaj Binkowski Ricardo Barreira Oriol Vinyals Andrew Zisserman Karen Simonyan MLLM VLM 46 3,349 0 29 Apr 2022
VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance Katherine Crowson Stella Biderman Daniel Kornis Dashiell Stander Eric Hallahan Louis Castricato Edward Raff CLIP 74 369 0 18 Apr 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 215 1,661 0 15 Oct 2021
Coarse-to-Fine Reasoning for Visual Question Answering Binh X. Nguyen Tuong Khanh Long Do Huy Tran Erman Tjiputra Quang-Dieu Tran A. Nguyen NAI 70 36 0 06 Oct 2021
xGQA: Cross-Lingual Visual Question Answering Jonas Pfeiffer Gregor Geigle Aishwarya Kamath Jan-Martin O. Steitz Stefan Roth Ivan Vulić Iryna Gurevych 37 56 0 13 Sep 2021
An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA Zhengyuan Yang Zhe Gan Jianfeng Wang Xiaowei Hu Yumao Lu Zicheng Liu Lijuan Wang 180 402 0 10 Sep 2021
How Much Can CLIP Benefit Vision-and-Language Tasks? Sheng Shen Liunian Harold Li Hao Tan Joey Tianyi Zhou Anna Rohrbach Kai-Wei Chang Z. Yao Kurt Keutzer CLIP VLM MLLM 202 405 0 13 Jul 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,858 0 18 Apr 2021
WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning Krishna Srinivasan K. Raman Jiecao Chen Michael Bendersky Marc Najork VLM 210 310 0 02 Mar 2021
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts Soravit Changpinyo P. Sharma Nan Ding Radu Soricut VLM 299 1,084 0 17 Feb 2021
High-Performance Large-Scale Image Recognition Without Normalization Andrew Brock Soham De Samuel L. Smith Karen Simonyan VLM 223 512 0 11 Feb 2021
Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision Chao Jia Yinfei Yang Ye Xia Yi-Ting Chen Zarana Parekh Hieu H. Pham Quoc V. Le Yun-hsuan Sung Zhen Li Tom Duerig VLM CLIP 337 3,708 0 11 Feb 2021
The Pile: An 800GB Dataset of Diverse Text for Language Modeling Leo Gao Stella Biderman Sid Black Laurence Golding Travis Hoppe ... Horace He Anish Thite Noa Nabeshima Shawn Presser Connor Leahy AIMat 282 2,000 0 31 Dec 2020