"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models

7 August 2023

Michael Backes

Papers citing ""Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models"

43 / 43 papers shown

Title
JALMBench: Benchmarking Jailbreak Vulnerabilities in Audio Language Models Zifan Peng Yule Liu Zhen Sun Mingchen Li Zeren Luo ... Xinlei He Xuechao Wang Yingjie Xue Shengmin Xu Xinyi Huang AuLLM AAML 26 0 0 23 May 2025
SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment Wonje Jeung Sangyeon Yoon Minsuk Kahng Albert No LRM LLMSV 84 1 0 20 May 2025
Safety Alignment Can Be Not Superficial With Explicit Safety Signals Jianwei Li Jung-Eng Kim AAML 57 0 0 19 May 2025
From Assistants to Adversaries: Exploring the Security Risks of Mobile LLM Agents Liangxuan Wu Chao Wang Tianming Liu Yanjie Zhao Haoyu Wang AAML 35 0 0 19 May 2025
Red Teaming the Mind of the Machine: A Systematic Evaluation of Prompt Injection and Jailbreak Vulnerabilities in LLMs Chetan Pathade AAML SILM 102 1 0 07 May 2025
ACE: A Security Architecture for LLM-Integrated App Systems Evan Li Tushin Mallick Evan Rose William K. Robertson Alina Oprea Cristina Nita-Rotaru 62 1 0 29 Apr 2025
Prefill-Based Jailbreak: A Novel Approach of Bypassing LLM Safety Boundary Yakai Li Jiekang Hu Weiduan Sang Luping Ma Jing Xie Weijuan Zhang Aimin Yu Shijie Zhao Qingjia Huang Qihang Zhou AAML 60 0 0 28 Apr 2025
DoomArena: A framework for Testing AI Agents Against Evolving Security Threats Léo Boisvert Mihir Bansal Chandra Kiran Reddy Evuru Gabriel Huang Abhay Puri ... Quentin Cappart Jason Stanley Alexandre Lacoste Alexandre Drouin Krishnamurthy Dvijotham 44 1 0 18 Apr 2025
Revealing the Intrinsic Ethical Vulnerability of Aligned Large Language Models Jiawei Lian Jianhong Pan L. Wang Yi Wang Shaohui Mei Lap-Pui Chau AAML 63 0 0 07 Apr 2025
$PiCo: Jailbreaking Multimodal Large Language Models via $\textbf{Pi}$ctorial $\textbf{Co}$de Contextualization$ PiCo: Jailbreaking Multimodal Large Language Models via $\textbf{Pi}$ ctorial $\textbf{Co}$ de Contextualization Aofan Liu Lulu Tang Ting Pan Yuguo Yin Bin Wang Ao Yang MLLM AAML 90 0 0 02 Apr 2025
Can LLMs Maintain Fundamental Abilities under KV Cache Compression? Xiang Liu Zhenheng Tang Hong Chen Peijie Dong Zeyu Li Xiuze Zhou Bo Li Xuming Hu Xiaowen Chu 313 5 0 04 Feb 2025
HateBench: Benchmarking Hate Speech Detectors on LLM-Generated Content and Hate Campaigns Xinyue Shen Yixin Wu Y. Qu Michael Backes Savvas Zannettou Yang Zhang 59 4 0 28 Jan 2025
Benchmarking and Defending Against Indirect Prompt Injection Attacks on Large Language Models Jingwei Yi Yueqi Xie Bin Zhu Emre Kiciman Guangzhong Sun Xing Xie Fangzhao Wu AAML 69 67 0 28 Jan 2025
When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Search Xuan Chen Yuzhou Nie Wenbo Guo Xiangyu Zhang 120 12 0 28 Jan 2025
LLM-Virus: Evolutionary Jailbreak Attack on Large Language Models Miao Yu Sihang Li Yingjie Zhou Xing Fan Kun Wang Shirui Pan Qingsong Wen AAML 90 1 0 03 Jan 2025
Position: A taxonomy for reporting and describing AI security incidents L. Bieringer Kevin Paeth Andreas Wespi Kathrin Grosse Alexandre Alahi Kathrin Grosse 110 0 0 19 Dec 2024
A Flexible Large Language Models Guardrail Development Methodology Applied to Off-Topic Prompt Detection Gabriel Chua Shing Yee Chan Shaun Khoo 133 1 0 20 Nov 2024
On the Role of Attention Heads in Large Language Model Safety Zhenhong Zhou Haiyang Yu Xinghua Zhang Rongwu Xu Fei Huang Kun Wang Yang Liu Sihang Li Yongbin Li 82 8 0 17 Oct 2024
AdvBDGen: Adversarially Fortified Prompt-Specific Fuzzy Backdoor Generator Against LLM Alignment Pankayaraj Pathmanathan Udari Madhushani Sehwag Michael-Andrei Panaitescu-Liess Furong Huang SILM AAML 58 0 0 15 Oct 2024
Bridging Today and the Future of Humanity: AI Safety in 2024 and Beyond Shanshan Han 106 1 0 09 Oct 2024
Permissive Information-Flow Analysis for Large Language Models Shoaib Ahmed Siddiqui Radhika Gaonkar Boris Köpf David M. Krueger Andrew Paverd Ahmed Salem Shruti Tople Lukas Wutschitz Menglin Xia Santiago Zanella Béguelin 59 1 0 04 Oct 2024
Robust LLM safeguarding via refusal feature adversarial training L. Yu Virginie Do Karen Hambardzumyan Nicola Cancedda AAML 73 14 0 30 Sep 2024
Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Data Atilla Akkus Mingjie Li Junjie Chu Junjie Chu Michael Backes Sinem Sav Sinem Sav SILM SyDa 60 3 0 12 Sep 2024
Differentially Private Kernel Density Estimation Erzhi Liu Jerry Yao-Chieh Hu Alex Reneau Zhao Song Han Liu 78 3 0 03 Sep 2024
Are Large Language Models Really Bias-Free? Jailbreak Prompts for Assessing Adversarial Robustness to Bias Elicitation Riccardo Cantini Giada Cosenza A. Orsino Domenico Talia AAML 67 6 0 11 Jul 2024
SelfDefend: LLMs Can Defend Themselves against Jailbreaking in a Practical Manner Xunguang Wang Daoyuan Wu Zhenlan Ji Zongjie Li Pingchuan Ma Shuai Wang Yingjiu Li Yang Liu Ning Liu Juergen Rahmel AAML 94 8 0 08 Jun 2024
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack M. Russinovich Ahmed Salem Ronen Eldan 68 86 0 02 Apr 2024
Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Models Yifan Li Hangyu Guo Kun Zhou Wayne Xin Zhao Ji-Rong Wen 72 45 0 14 Mar 2024
GUARD: Role-playing to Generate Natural-language Jailbreakings to Test Guideline Adherence of Large Language Models Haibo Jin Ruoxi Chen Peiyan Zhang Andy Zhou Yang Zhang Haohan Wang LLMAG 30 23 0 05 Feb 2024
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs Yi Zeng Hongpeng Lin Jingwen Zhang Diyi Yang Ruoxi Jia Weiyan Shi 41 284 0 12 Jan 2024
FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts Yichen Gong Delong Ran Jinyuan Liu Conglei Wang Tianshuo Cong Anyu Wang Sisi Duan Xiaoyun Wang MLLM 162 138 0 09 Nov 2023
Demystifying RCE Vulnerabilities in LLM-Integrated Apps Tong Liu Zizhuang Deng Guozhu Meng Yuekang Li Kai Chen SILM 73 19 0 06 Sep 2023
Universal and Transferable Adversarial Attacks on Aligned Language Models Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J. Zico Kolter Matt Fredrikson 139 1,376 0 27 Jul 2023
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models Wei Ping Weixin Chen Hengzhi Pei Chulin Xie Mintong Kang ... Zinan Lin Yuk-Kit Cheng Sanmi Koyejo D. Song Yue Liu 36 405 0 20 Jun 2023
On the Reliability of Watermarks for Large Language Models John Kirchenbauer Jonas Geiping Yuxin Wen Manli Shu Khalid Saifullah Kezhi Kong Kasun Fernando Aniruddha Saha Micah Goldblum Tom Goldstein WaLM 32 113 0 07 Jun 2023
Analyzing Leakage of Personally Identifiable Information in Language Models Nils Lukas A. Salem Robert Sim Shruti Tople Lukas Wutschitz Santiago Zanella Béguelin PILM 65 218 0 01 Feb 2023
Bad Characters: Imperceptible NLP Attacks Nicholas Boucher Ilia Shumailov Ross J. Anderson Nicolas Papernot AAML SILM 49 104 0 18 Jun 2021
GLM: General Language Model Pretraining with Autoregressive Blank Infilling Zhengxiao Du Yujie Qian Xiao Liu Ming Ding J. Qiu Zhilin Yang Jie Tang BDL AI4CE 69 1,520 0 18 Mar 2021
Extracting Training Data from Large Language Models Nicholas Carlini Florian Tramèr Eric Wallace Matthew Jagielski Ariel Herbert-Voss ... Tom B. Brown D. Song Ulfar Erlingsson Alina Oprea Colin Raffel MLAU SILM 373 1,868 0 14 Dec 2020
Beyond Accuracy: Behavioral Testing of NLP models with CheckList Marco Tulio Ribeiro Tongshuang Wu Carlos Guestrin Sameer Singh ELM 98 1,089 0 08 May 2020
Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks Nils Reimers Iryna Gurevych 456 11,979 0 27 Aug 2019
Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment Di Jin Zhijing Jin Qiufeng Wang Peter Szolovits SILM AAML 74 1,064 0 27 Jul 2019
Adversarial Example Generation with Syntactically Controlled Paraphrase Networks Mohit Iyyer John Wieting Kevin Gimpel Luke Zettlemoyer AAML GAN 291 715 0 17 Apr 2018