Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation

14 March 2024

Yunhao Gou

Kai Chen

Zhili Liu

Lanqing Hong

Hang Xu

Papers citing "Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation"

22 / 22 papers shown

Title
Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects Agnese Chiatti Sara Bernardini Lara Shibelski Godoy Piccolo Viola Schiaffonati Matteo Matteucci 62 0 0 08 May 2025
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models J. Liu Hangyu Guo Ranjie Duan Xingyuan Bu Yancheng He ... Yingshui Tan Yanan Wu Jihao Gu Y. Li J. Zhu MLLM 145 0 0 25 Apr 2025
CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models Xiangyu Yin Jiaxu Liu Zhen Chen Jinwei Hu Yi Dong Xiaowei Huang Wenjie Ruan AAML 47 0 0 08 Mar 2025
Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment Zhili Liu Yunhao Gou Kai Chen Lanqing Hong Jiahui Gao ... Yu Zhang Zhenguo Li Xin Jiang Q. Liu James T. Kwok MoE 96 9 0 20 Feb 2025
Understanding and Rectifying Safety Perception Distortion in VLMs Xiaohan Zou Jian Kang George Kesidis Lu Lin 169 1 0 18 Feb 2025
UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models Sejoon Oh Yiqiao Jin Megha Sharma Donghyun Kim Eric Ma Gaurav Verma Srijan Kumar 60 6 0 03 Nov 2024
Cross-Modal Safety Mechanism Transfer in Large Vision-Language Models Shicheng Xu Liang Pang Yunchang Zhu Huawei Shen Xueqi Cheng MLLM 36 1 0 16 Oct 2024
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time Yi Ding Bolian Li Ruqi Zhang MLLM 72 6 0 09 Oct 2024
GLOV: Guided Large Language Models as Implicit Optimizers for Vision Language Models Muhammad Jehanzeb Mirza Mengjie Zhao Zhuoyuan Mao Sivan Doveh Wei Lin ... Yuki Mitsufuji Horst Possegger Rogerio Feris Leonid Karlinsky James Glass VLM 84 1 0 08 Oct 2024
EMOVA: Empowering Language Models to See, Hear and Speak with Vivid Emotions Kai Chen Yunhao Gou Runhui Huang Zhili Liu Daxin Tan ... Qun Liu Jun Yao Lu Hou Hang Xu Hang Xu AuLLM MLLM VLM 82 21 0 26 Sep 2024
BaThe: Defense against the Jailbreak Attack in Multimodal Large Language Models by Treating Harmful Instruction as Backdoor Trigger Yulin Chen Haoran Li Zihao Zheng Zihao Zheng Yangqiu Song Bryan Hooi 50 6 0 17 Aug 2024
DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception Yibo Wang Ruiyuan Gao Kai Chen Kaiqiang Zhou Yingjie Cai ... Zhenguo Li Lihui Jiang Dit-Yan Yeung Qiang Xu Kai Zhang DiffM 118 21 0 20 Mar 2024
Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts Zhili Liu Kai Chen Jianhua Han Lanqing Hong Hang Xu Zhenguo Li James T. Kwok MoE 111 24 0 08 Feb 2024
Safety Fine-Tuning at (Almost) No Cost: A Baseline for Vision Large Language Models Yongshuo Zong Ondrej Bohdal Tingyang Yu Yongxin Yang Timothy M. Hospedales VLM MLLM 54 56 0 03 Feb 2024
Red Teaming Visual Language Models Mukai Li Lei Li Yuwei Yin Masood Ahmed Zhenguang Liu Qi Liu VLM 36 30 0 23 Jan 2024
V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs Penghao Wu Saining Xie LRM 60 122 0 21 Dec 2023
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts Yichen Gong Delong Ran Jinyuan Liu Conglei Wang Tianshuo Cong Anyu Wang Sisi Duan Xiaoyun Wang MLLM 129 118 0 09 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 126 375 0 07 Nov 2023
On the Adversarial Robustness of Multi-Modal Foundation Models Christian Schlarmann Matthias Hein AAML 116 85 0 21 Aug 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 211 1,106 0 20 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong S. Hoi MLLM BDL VLM CLIP 392 4,137 0 28 Jan 2022