Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery

7 February 2023

Papers citing "Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery"

47 / 197 papers shown

Title
Towards image compression with perfect realism at ultra-low bitrates Marlene Careil Matthew Muckley Jakob Verbeek Stéphane Lathuilière DiffM 32 46 0 16 Oct 2023
Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models? Yu-Lin Tsai Chia-Yi Hsu Chulin Xie Chih-Hsun Lin Jia-You Chen Bo-wen Li Pin-Yu Chen Chia-Mu Yu Chun-ying Huang DiffM 44 78 0 16 Oct 2023
Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation Yangsibo Huang Samyak Gupta Mengzhou Xia Kai Li Danqi Chen AAML 35 276 0 10 Oct 2023
Teaching Language Models to Hallucinate Less with Synthetic Tasks Erik Jones Hamid Palangi Clarisse Simoes Varun Chandrasekaran Subhabrata Mukherjee Arindam Mitra Ahmed Hassan Awadallah Ece Kamar HILM 26 24 0 10 Oct 2023
Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations Zeming Wei Yifei Wang Ang Li Yichuan Mo Yisen Wang 54 239 0 10 Oct 2023
Understanding prompt engineering may not require rethinking generalization Victor Akinwande Yiding Jiang Dylan Sam J. Zico Kolter VLM VPVLM 123 7 0 06 Oct 2023
Navigating Cultural Chasms: Exploring and Unlocking the Cultural POV of Text-To-Image Models Mor Ventura Eyal Ben-David Anna Korhonen Roi Reichart 21 11 0 03 Oct 2023
Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models Zijun Wu Yongkang Wu Lili Mou VLM 35 2 0 02 Oct 2023
What's the Magic Word? A Control Theory of LLM Prompting Aman Bhargava Cameron Witkowski Manav Shah Matt W. Thomson LLMAG 61 30 0 02 Oct 2023
Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts Zhi-Yi Chin Chieh-Ming Jiang Ching-Chun Huang Pin-Yu Chen Wei-Chen Chiu DiffM 24 67 0 12 Sep 2023
Large Language Models as Optimizers Chengrun Yang Xuezhi Wang Yifeng Lu Hanxiao Liu Quoc V. Le Denny Zhou Xinyun Chen ODL 43 379 0 07 Sep 2023
Baseline Defenses for Adversarial Attacks Against Aligned Language Models Neel Jain Avi Schwarzschild Yuxin Wen Gowthami Somepalli John Kirchenbauer Ping Yeh-Chiang Micah Goldblum Aniruddha Saha Jonas Geiping Tom Goldstein AAML 60 343 0 01 Sep 2023
Manipulating Embeddings of Stable Diffusion Prompts Niklas Deckers Julia Peters Martin Potthast DiffM 42 9 0 23 Aug 2023
Reverse Stable Diffusion: What prompt was used to generate this image? Florinel-Alin Croitoru Vlad Hondru Radu Tudor Ionescu M. Shah VLM DiffM 44 6 0 02 Aug 2023
ImageBrush: Learning Visual In-Context Instructions for Exemplar-Based Image Manipulation Yasheng Sun Yifan Yang Houwen Peng Yifei Shen Yuqing Yang Hang-Rui Hu Lili Qiu Hideki Koike DiffM LM&Ro 42 33 0 02 Aug 2023
Guiding Image Captioning Models Toward More Specific Captions Simon Kornblith Lala Li Zirui Wang Thao Nguyen 32 15 0 31 Jul 2023
Universal and Transferable Adversarial Attacks on Aligned Language Models Andy Zou Zifan Wang Nicholas Carlini Milad Nasr J. Zico Kolter Matt Fredrikson 109 1,289 0 27 Jul 2023
Visual Instruction Inversion: Image Editing via Visual Prompting Thao Nguyen Yuheng Li Utkarsh Ojha Yong Jae Lee DiffM 32 22 0 26 Jul 2023
AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models Jiachun Pan Jun Hao Liew Vincent Y. F. Tan Jiashi Feng Hanshu Yan DiffM 32 10 0 20 Jul 2023
Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models Zhiyuan Peng Xuyang Wu Qifan Wang Yihan Fang VLM RALM 54 11 0 17 Jul 2023
Personalization for BERT-based Discriminative Speech Recognition Rescoring J. Kolehmainen Yile Gu Aditya Gourav Prashanth Gurunath Shivakumar Ankur Gandhe Ariya Rastrow I. Bulyko 38 2 0 13 Jul 2023
Text Descriptions are Compressive and Invariant Representations for Visual Learning Zhili Feng Anna Bair J. Zico Kolter VLM 24 6 0 10 Jul 2023
Text + Sketch: Image Compression at Ultra Low Rates Eric Lei Yiugit Berkay Uslu Hamed Hassani Shirin Saeedi Bidokhti DiffM 23 38 0 04 Jul 2023
Mass-Producing Failures of Multimodal Systems with Language Models Shengbang Tong Erik Jones Jacob Steinhardt 49 34 0 21 Jun 2023
Explore, Establish, Exploit: Red Teaming Language Models from Scratch Stephen Casper Jason Lin Joe Kwon Gatlen Culp Dylan Hadfield-Menell AAML 21 85 0 15 Jun 2023
User-friendly Image Editing with Minimal Text Input: Leveraging Captioning and Injection Techniques Sunwoo Kim Wooseok Jang Hyunsung Kim Junho Kim Yunjey Choi Seung Wook Kim Gayeong Lee DiffM 47 6 0 05 Jun 2023
StyleDrop: Text-to-Image Generation in Any Style Kihyuk Sohn Nataniel Ruiz Kimin Lee Daniel Castro Chin Irina Blok ... Yuanzhen Li Yuan Hao Irfan Essa Michael Rubinstein Dilip Krishnan 11 143 0 01 Jun 2023
Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search Qihao Liu Adam Kortylewski Yutong Bai Song Bai Alan Yuille DiffM 40 12 0 01 Jun 2023
The Hidden Language of Diffusion Models Hila Chefer Oran Lang Mor Geva Volodymyr Polosukhin Assaf Shocher Michal Irani Inbar Mosseri Lior Wolf DiffM 30 26 0 01 Jun 2023
Understanding and Mitigating Copying in Diffusion Models Gowthami Somepalli Vasu Singla Micah Goldblum Jonas Geiping Tom Goldstein DiffM 20 125 0 31 May 2023
Submodular Minimax Optimization: Finding Effective Sets Loay Mualem Ethan R. Elenberg Moran Feldman Amin Karbasi 28 5 0 26 May 2023
HUB: Guiding Learned Optimizers with Continuous Prompt Tuning Gaole Dai Wei Wu Ziyu Wang Jie Fu Shanghang Zhang Tiejun Huang AIFin 16 0 0 26 May 2023
ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Models Yuxin Zhang Weiming Dong Fan Tang Nisha Huang Haibin Huang Chongyang Ma Tong-Yee Lee Oliver Deussen Changsheng Xu DiffM 40 75 0 25 May 2023
Universal Self-Adaptive Prompting Xingchen Wan Ruoxi Sun Hootan Nakhost H. Dai Julian Martin Eisenschlos Sercan Ö. Arik Tomas Pfister LRM 38 9 0 24 May 2023
Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models Stephen Brade Bryan Wang Maurício Sousa Sageev Oore Tovi Grossman MLLM DiffM 32 83 0 18 Apr 2023
Expressive Text-to-Image Generation with Rich Text Songwei Ge Taesung Park Jun-Yan Zhu Jia-Bin Huang DiffM 79 78 0 13 Apr 2023
A Pilot Study of Query-Free Adversarial Attack against Stable Diffusion Haomin Zhuang Yihua Zhang Sijia Liu DiffM 45 62 0 29 Mar 2023
Zeroth-Order Optimization Meets Human Feedback: Provable Learning via Ranking Oracles Zhiwei Tang Dmitry Rybin Tsung-Hui Chang ALM DiffM 39 26 0 07 Mar 2023
Collage Diffusion Vishnu Sarukkai Linden Li Arden Ma Christopher Ré Kayvon Fatahalian DiffM 31 24 0 01 Mar 2023
Glaze: Protecting Artists from Style Mimicry by Text-to-Image Models Shawn Shan Jenna Cryan Emily Wenger Haitao Zheng Rana Hanocka Ben Y. Zhao WIGM 17 177 0 08 Feb 2023
Discovering and Mitigating Visual Biases through Keyword Explanation Younghyun Kim Sangwoo Mo Minkyu Kim Kyungmin Lee Jaeho Lee Jinwoo Shin 40 33 0 26 Jan 2023
Red-Teaming the Stable Diffusion Safety Filter Javier Rando Daniel Paleka David Lindner Lennard Heim Florian Tramèr DiffM 132 184 0 03 Oct 2022
Discovering the Hidden Vocabulary of DALLE-2 Giannis Daras A. Dimakis 132 64 0 01 Jun 2022
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation Junnan Li Dongxu Li Caiming Xiong Guosheng Lin MLLM BDL VLM CLIP 392 4,171 0 28 Jan 2022
Multitask Prompted Training Enables Zero-Shot Task Generalization Victor Sanh Albert Webson Colin Raffel Stephen H. Bach Lintang Sutawika ... T. Bers Stella Biderman Leo Gao Thomas Wolf Alexander M. Rush LRM 218 1,663 0 15 Oct 2021
The Power of Scale for Parameter-Efficient Prompt Tuning Brian Lester Rami Al-Rfou Noah Constant VPVLM 280 3,872 0 18 Apr 2021
Gradient-based Adversarial Attacks against Text Transformers Chuan Guo Alexandre Sablayrolles Hervé Jégou Douwe Kiela SILM 106 228 0 15 Apr 2021