On the Adversarial Robustness of Multi-Modal Foundation Models

21 August 2023

Matthias Hein

Papers citing "On the Adversarial Robustness of Multi-Modal Foundation Models"

50 / 64 papers shown

Title
X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP Hanxun Huang Sarah Monazam Erfani Yige Li Xingjun Ma James Bailey AAML 44 0 0 08 May 2025
Transferable Adversarial Attacks on Black-Box Vision-Language Models Kai Hu Weichen Yu L. Zhang Alexander Robey Andy Zou Chengming Xu Haoqi Hu Matt Fredrikson AAML VLM 57 0 0 02 May 2025
Multimodal Large Language Models for Enhanced Traffic Safety: A Comprehensive Review and Future Trends M. Tami Mohammed Elhenawy Huthaifa I. Ashqar 31 0 0 21 Apr 2025
A Domain-Based Taxonomy of Jailbreak Vulnerabilities in Large Language Models Carlos Peláez-González Andrés Herrera-Poyatos Cristina Zuheros David Herrera-Poyatos Virilo Tejedor F. Herrera AAML 21 0 0 07 Apr 2025
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization Chaohu Liu Tianyi Gui Yu Liu Linli Xu VLM AAML 68 1 0 02 Apr 2025
Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models Runpeng Dai Run Yang Fan Zhou Hongtu Zhu 28 0 0 28 Mar 2025
Survey of Adversarial Robustness in Multimodal Large Language Models Chengze Jiang Zhuangzhuang Wang Minjing Dong Jie Gui AAML 60 0 0 18 Mar 2025
Evolution-based Region Adversarial Prompt Learning for Robustness Enhancement in Vision-Language Models X. Jia Sensen Gao Simeng Qin Ke Ma X. Li Yihao Huang Wei Dong Yang Liu Xiaochun Cao AAML VLM 60 0 0 17 Mar 2025
Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalization Shuyang Hao Yiwei Wang Bryan Hooi J. Liu Muhao Chen Zi Huang Yujun Cai AAML VLM 67 0 0 14 Mar 2025
Robustness Tokens: Towards Adversarial Robustness of Transformers Brian Pulfer Yury Belousov S. Voloshynovskiy AAML 45 0 0 13 Mar 2025
Poisoned-MRAG: Knowledge Poisoning Attacks to Multimodal Retrieval Augmented Generation Yinuo Liu Zenghui Yuan Guiyao Tie Jiawen Shi Lichao Sun Lichao Sun Neil Zhenqiang Gong 40 1 0 08 Mar 2025
MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks Hyeonjeong Ha Qiusi Zhan Jeonghwan Kim Dimitrios Bralios Saikrishna Sanniboina Nanyun Peng Kai-Wei Chang Daniel Kang Heng Ji KELM AAML 67 1 0 25 Feb 2025
Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images Yubo Wang Jianting Tang Chaohu Liu Linli Xu AAML 61 1 0 23 Feb 2025
Natural Language Generation from Visual Sequences: Challenges and Future Directions Aditya K Surikuchi Raquel Fernández Sandro Pezzelle EGVM 197 0 0 18 Feb 2025
On Creating a Causally Grounded Usable Rating Method for Assessing the Robustness of Foundation Models Supporting Time Series Kausik Lakkaraju Rachneet Kaur Parisa Zehtabi Sunandita Patra Siva Likitha Valluru Zhen Zeng Biplav Srivastava Marco Valtorta AI4TS 82 0 0 17 Feb 2025
Robust-LLaVA: On the Effectiveness of Large-Scale Robust Image Encoders for Multi-modal Large Language Models H. Malik Fahad Shamshad Muzammal Naseer Karthik Nandakumar F. Khan Salman Khan AAML MLLM VLM 68 0 0 03 Feb 2025
Visual Large Language Models for Generalized and Specialized Applications Yifan Li Zhixin Lai Wentao Bao Zhen Tan Anh Dao Kewei Sui Jiayi Shen Dong Liu Huan Liu Yu Kong VLM 88 11 0 06 Jan 2025
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models Shuyang Hao Bryan Hooi J. Liu Kai-Wei Chang Zi Huang Yujun Cai AAML 92 0 0 27 Nov 2024
Rethinking the Intermediate Features in Adversarial Attacks: Misleading Robotic Models via Adversarial Distillation Ke Zhao Huayang Huang Miao Li Yu Wu AAML 71 0 0 21 Nov 2024
SoK: Unifying Cybersecurity and Cybersafety of Multimodal Foundation Models with an Information Theory Approach Ruoxi Sun Jiamin Chang Hammond Pearce Chaowei Xiao B. Li Qi Wu Surya Nepal Minhui Xue 35 0 0 17 Nov 2024
New Emerged Security and Privacy of Pre-trained Model: a Survey and Outlook Meng Yang Tianqing Zhu Chi Liu Wanlei Zhou Shui Yu Philip S. Yu AAML ELM PILM 61 1 0 12 Nov 2024
Unfair Alignment: Examining Safety Alignment Across Vision Encoder Layers in Vision-Language Models Saketh Bachu Erfan Shayegani Trishna Chakraborty Rohit Lal Arindam Dutta Chengyu Song Yue Dong Nael B. Abu-Ghazaleh A. Roy-Chowdhury 34 0 0 06 Nov 2024
Order Matters: Exploring Order Sensitivity in Multimodal Large Language Models Zhijie Tan Xu Chu Weiping Li Tong Mo 28 1 0 22 Oct 2024
Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models Yubo Wang Chaohu Liu Yanqiu Qu Haoyu Cao Deqiang Jiang Linli Xu MLLM AAML 29 3 0 09 Oct 2024
R-Bench: Are your Large Multimodal Model Robust to Real-world Corruptions? Chunyi Li J. Zhang Zicheng Zhang H. Wu Yuan Tian ... Guo Lu Xiaohong Liu Xiongkuo Min Weisi Lin Guangtao Zhai AAML 39 3 0 07 Oct 2024
HiddenGuard: Fine-Grained Safe Generation with Specialized Representation Router Lingrui Mei Shenghua Liu Yiwei Wang Baolong Bi Ruibin Yuan Xueqi Cheng 35 4 0 03 Oct 2024
Enhancing LLM-based Autonomous Driving Agents to Mitigate Perception Attacks Ruoyu Song Muslum Ozgur Ozmen Hyungsub Kim Antonio Bianchi Z. Berkay Celik AAML 28 5 0 22 Sep 2024
Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks Md Zarif Hossain Ahmed Imteaj AAML VLM 40 3 0 11 Sep 2024
BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger Yulin Chen Haoran Li Zihao Zheng Zihao Zheng Yangqiu Song Bryan Hooi 45 6 0 17 Aug 2024
BAPLe: Backdoor Attacks on Medical Foundational Models using Prompt Learning Asif Hanif Fahad Shamshad Muhammad Awais Muzammal Naseer F. Khan Karthik Nandakumar Salman Khan Rao Muhammad Anwer MedIm AAML 40 3 0 14 Aug 2024
AI Safety in Practice: Enhancing Adversarial Robustness in Multimodal Image Captioning Maisha Binte Rashid Pablo Rivas 19 2 0 30 Jul 2024
Sparse vs Contiguous Adversarial Pixel Perturbations in Multimodal Models: An Empirical Analysis Cristian-Alexandru Botocan Raphael Meier Ljiljana Dolamic AAML 19 0 0 25 Jul 2024
Sim-CLIP: Unsupervised Siamese Adversarial Fine-Tuning for Robust and Semantically-Rich Vision-Language Models Md Zarif Hossain Ahmed Imteaj VLM AAML 39 4 0 20 Jul 2024
Refusing Safe Prompts for Multi-modal Large Language Models Zedian Shao Hongbin Liu Yuepeng Hu Neil Zhenqiang Gong MLLM LRM 41 1 0 12 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 57 5 0 11 Jul 2024
A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends Daizong Liu Mingyu Yang Xiaoye Qu Pan Zhou Yu Cheng Wei Hu ELM AAML 30 25 0 10 Jul 2024
JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models Haibo Jin Leyang Hu Xinuo Li Peiyan Zhang Chonghan Chen Jun Zhuang Haohan Wang PILM 36 26 0 26 Jun 2024
Texture Re-scalable Universal Adversarial Perturbation Yihao Huang Qing Guo Felix Juefei-Xu Ming Hu Xiaojun Jia Xiaochun Cao Geguang Pu Yang Liu AAML 27 7 0 10 Jun 2024
Improving Alignment and Robustness with Circuit Breakers Andy Zou Long Phan Justin Wang Derek Duenas Maxwell Lin Maksym Andriushchenko Rowan Wang Zico Kolter Matt Fredrikson Dan Hendrycks AAML 39 71 0 06 Jun 2024
Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt Zonghao Ying Aishan Liu Tianyuan Zhang Zhengmin Yu Siyuan Liang Xianglong Liu Dacheng Tao AAML 33 26 0 06 Jun 2024
White-box Multimodal Jailbreaks Against Large Vision-Language Models Ruofan Wang Xingjun Ma Hanxu Zhou Chuanjun Ji Guangnan Ye Yu-Gang Jiang AAML VLM 41 17 0 28 May 2024
Cross-Modal Safety Alignment: Is textual unlearning all you need? Trishna Chakraborty Erfan Shayegani Zikui Cai Nael B. Abu-Ghazaleh M. Salman Asif Yue Dong A. Roy-Chowdhury Chengyu Song 39 15 0 27 May 2024
Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning Zihua Zhao Mengxi Chen Tianjie Dai Jiangchao Yao Bo han Ya-Qin Zhang Yanfeng Wang NoLa 34 3 0 27 May 2024
Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character Siyuan Ma Weidi Luo Yu Wang Xiaogeng Liu 35 20 0 25 May 2024
Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security Yihe Fan Yuxin Cao Ziyu Zhao Ziyao Liu Shaofeng Li 30 12 0 08 Apr 2024
Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models Yifan Li Hangyu Guo Kun Zhou Wayne Xin Zhao Ji-Rong Wen 55 38 0 14 Mar 2024
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation Yunhao Gou Kai Chen Zhili Liu Lanqing Hong Hang Xu Zhenguo Li Dit-Yan Yeung James T. Kwok Yu Zhang MLLM 37 37 0 14 Mar 2024
One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models Lin Li Haoyan Guan Jianing Qiu Michael W. Spratling AAML VLM VPVLM 31 21 0 04 Mar 2024
Robust CLIP: Unsupervised Adversarial Fine-Tuning of Vision Embeddings for Robust Large Vision-Language Models Christian Schlarmann Naman D. Singh Francesco Croce Matthias Hein VLM AAML 47 37 0 19 Feb 2024
Test-Time Backdoor Attacks on Multimodal Large Language Models Dong Lu Tianyu Pang Chao Du Qian Liu Xianjun Yang Min-Bin Lin AAML 53 21 0 13 Feb 2024