DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback

16 November 2023

Heng Ji

Papers citing "DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedback"

50 / 62 papers shown

Title
Exploring Implicit Visual Misunderstandings in Multimodal Large Language Models through Attention Analysis Pengfei Wang Guohai Xu Weinong Wang Junjie Yang Jie Lou Yunhua Xue 24 0 0 15 May 2025
Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training Y. Chen Hao Peng Tong Zhang Heng Ji VLM 28 0 0 13 May 2025
Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospects Agnese Chiatti Sara Bernardini Lara Shibelski Godoy Piccolo Viola Schiaffonati Matteo Matteucci 62 0 0 08 May 2025
Sailing AI by the Stars: A Survey of Learning from Rewards in Post-Training and Test-Time Scaling of Large Language Models Xiaobao Wu LRM 72 1 0 05 May 2025
DREAM: Disentangling Risks to Enhance Safety Alignment in Multimodal Large Language Models J. Liu Hangyu Guo Ranjie Duan Xingyuan Bu Yancheng He ... Yingshui Tan Yanan Wu Jihao Gu Y. Li J. Zhu MLLM 142 0 0 25 Apr 2025
Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-tuning Yiwei Chen Yuguang Yao Yihua Zhang Bingquan Shen Gaowen Liu Sijia Liu AAML MU 58 1 0 14 Mar 2025
Tit-for-Tat: Safeguarding Large Vision-Language Models Against Jailbreak Attacks via Adversarial Defense Shuyang Hao Y. Wang Bryan Hooi Ming Yang J. Liu Chengcheng Tang Zi Huang Yujun Cai AAML 52 0 0 14 Mar 2025
TAIJI: Textual Anchoring for Immunizing Jailbreak Images in Vision Language Models Xiangyu Yin Yi Qi Jinwei Hu Zhen Chen Yi Dong Xingyu Zhao Xiaowei Huang Wenjie Ruan 45 0 0 13 Mar 2025
CeTAD: Towards Certified Toxicity-Aware Distance in Vision Language Models Xiangyu Yin Jiaxu Liu Zhen Chen Jinwei Hu Yi Dong Xiaowei Huang Wenjie Ruan AAML 45 0 0 08 Mar 2025
Understanding and Rectifying Safety Perception Distortion in VLMs Xiaohan Zou Jian Kang George Kesidis Lu Lin 166 1 0 18 Feb 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Y. Wang Chengxuan Qian Huaxiu Yao Zhengzhong Tu 94 6 0 18 Feb 2025
Towards Robust Multimodal Large Language Models Against Jailbreak Attacks Ziyi Yin Yuanpu Cao Han Liu Ting Wang Jinghui Chen Fenhlong Ma AAML 50 0 0 02 Feb 2025
Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection Le Yang Ziwei Zheng Boxu Chen Zhengyu Zhao Chenhao Lin Chao Shen VLM 140 3 0 18 Dec 2024
Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization Weiyun Wang Zhe Chen Wenhai Wang Yue Cao Yangzhou Liu ... Jinguo Zhu X. Zhu Lewei Lu Yu Qiao Jifeng Dai LRM 62 46 1 15 Nov 2024
GiVE: Guiding Visual Encoder to Perceive Overlooked Information Junjie Li Jianghong Ma Xiaofeng Zhang Yuhang Li Jianyang Shi 33 0 0 26 Oct 2024
Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment Chenhang Cui An Zhang Yiyang Zhou Zhaorun Chen Gelei Deng Huaxiu Yao Tat-Seng Chua 68 4 0 18 Oct 2024
VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment Lei Li Zhihui Xie Mukai Li Shunian Chen Peiyi Wang L. Chen Yazheng Yang Benyou Wang Lingpeng Kong Q. Liu VLM ALM 34 17 0 12 Oct 2024
Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models Qin Liu Chao Shang Ling Liu Nikolaos Pappas Jie Ma Neha Anna John Srikanth Doss Kadarundalagi Raghuram Doss Lluís Marquez Miguel Ballesteros Yassine Benajiba 34 4 0 11 Oct 2024
ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time Yi Ding Bolian Li Ruqi Zhang MLLM 72 5 0 09 Oct 2024
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks Jiayi He Hehai Lin Q. Wang Yi Ren Fung Heng Ji ReLM LRM 101 4 0 05 Oct 2024
Alignment of Diffusion Models: Fundamentals, Challenges, and Future Buhua Liu Shitong Shao Bao Li Lichen Bai Zhiqiang Xu Haoyi Xiong James Kwok Sumi Helal Zeke Xie 42 12 0 11 Sep 2024
Defending Jailbreak Attack in VLMs via Cross-modality Information Detector Yue Xu Xiuyuan Qi Zhan Qin Wenjie Wang AAML 45 6 0 31 Jul 2024
FIRE: A Dataset for Feedback Integration and Refinement Evaluation of Multimodal Models Pengxiang Li Zhi Gao Bofei Zhang Tao Yuan Yuwei Wu Mehrtash Harandi Yunde Jia Song-Chun Zhu Qing Li VLM MLLM 40 3 0 16 Jul 2024
The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective Zhen Qin Daoyuan Chen Wenhao Zhang Liuyi Yao Yilun Huang Bolin Ding Yaliang Li Shuiguang Deng 57 5 0 11 Jul 2024
A Survey of Attacks on Large Vision-Language Models: Resources, Advances, and Future Trends Daizong Liu Mingyu Yang Xiaoye Qu Pan Zhou Yu Cheng Wei Hu ELM AAML 30 25 0 10 Jul 2024
A Single Transformer for Scalable Vision-Language Modeling Yangyi Chen Xingyao Wang Hao Peng Heng Ji LRM 40 14 0 08 Jul 2024
Understanding Alignment in Multimodal LLMs: A Comprehensive Study Elmira Amirloo J. Fauconnier Christoph Roesmann Christian Kerl Rinu Boney ... Zirui Wang Afshin Dehghan Yinfei Yang Zhe Gan Peter Grasch 35 6 0 02 Jul 2024
Pelican: Correcting Hallucination in Vision-LLMs via Claim Decomposition and Program of Thought Verification Pritish Sahu Karan Sikka Ajay Divakaran MLLM LRM 66 4 0 02 Jul 2024
JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Models Haibo Jin Leyang Hu Xinuo Li Peiyan Zhang Chonghan Chen Jun Zhuang Haohan Wang PILM 36 26 0 26 Jun 2024
Evaluating and Analyzing Relationship Hallucinations in Large Vision-Language Models Mingrui Wu Jiayi Ji Oucheng Huang Jiale Li Yuhang Wu Xiaoshuai Sun Rongrong Ji 48 8 0 24 Jun 2024
SPA-VL: A Comprehensive Safety Preference Alignment Dataset for Vision Language Model Yongting Zhang Lu Chen Guodong Zheng Yifeng Gao Rui Zheng ... Yu Qiao Xuanjing Huang Feng Zhao Tao Gui Jing Shao VLM 79 23 0 17 Jun 2024
Jailbreak Vision Language Models via Bi-Modal Adversarial Prompt Zonghao Ying Aishan Liu Tianyuan Zhang Zhengmin Yu Siyuan Liang Xianglong Liu Dacheng Tao AAML 33 26 0 06 Jun 2024
A Survey of Multimodal Large Language Model from A Data-centric Perspective Tianyi Bai Hao Liang Binwang Wan Yanran Xu Xi Li ... Ping-Chia Huang Jiulong Shan Conghui He Binhang Yuan Wentao Zhang 47 36 0 26 May 2024
Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character Siyuan Ma Weidi Luo Yu Wang Xiaogeng Liu 35 20 0 25 May 2024
Unbridled Icarus: A Survey of the Potential Perils of Image Inputs in Multimodal Large Language Model Security Yihe Fan Yuxin Cao Ziyu Zhao Ziyao Liu Shaofeng Li 30 12 0 08 Apr 2024
FACTOID: FACtual enTailment fOr hallucInation Detection Vipula Rawte S. M. Towhidul Krishnav Rajbangshi Shravani Nag Aman Chadha Amit P. Sheth Amitava Das HILM 31 3 0 28 Mar 2024
Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformation Yunhao Gou Kai Chen Zhili Liu Lanqing Hong Hang Xu Zhenguo Li Dit-Yan Yeung James T. Kwok Yu Zhang MLLM 37 37 0 14 Mar 2024
The First to Know: How Token Distributions Reveal Hidden Knowledge in Large Vision-Language Models? Qinyu Zhao Ming Xu Kartik Gupta Akshay Asthana Liang Zheng Stephen Gould 29 7 0 14 Mar 2024
Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions Akash Ghosh Arkadeep Acharya Sriparna Saha Vinija Jain Aman Chadha VLM 57 25 0 20 Feb 2024
The Revolution of Multimodal Large Language Models: A Survey Davide Caffagni Federico Cocchi Luca Barsellotti Nicholas Moratelli Sara Sarto Lorenzo Baraldi Lorenzo Baraldi Marcella Cornia Rita Cucchiara LRM VLM 53 41 0 19 Feb 2024
Recursive Chain-of-Feedback Prevents Performance Degradation from Redundant Prompting Jinwoo Ahn Kyuseung Shin ReLM LRM AI4CE 13 1 0 05 Feb 2024
Executable Code Actions Elicit Better LLM Agents Xingyao Wang Yangyi Chen Lifan Yuan Yizhe Zhang Yunzhu Li Hao Peng Heng Ji ELM LLMAG LM&Ro 32 127 0 01 Feb 2024
Safety of Multimodal Large Language Models on Images and Texts Xin Liu Yichen Zhu Yunshi Lan Chao Yang Yu Qiao 24 28 0 01 Feb 2024
MM-LLMs: Recent Advances in MultiModal Large Language Models Duzhen Zhang Yahan Yu Jiahua Dong Chenxing Li Dan Su Chenhui Chu Dong Yu OffRL LRM 42 178 0 24 Jan 2024
A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models S.M. Towhidul Islam Tonmoy S. M. M. Zaman Vinija Jain Anku Rani Vipula Rawte Aman Chadha Amitava Das HILM 32 179 0 02 Jan 2024
MM-SafetyBench: A Benchmark for Safety Evaluation of Multimodal Large Language Models Xin Liu Yichen Zhu Jindong Gu Yunshi Lan Chao Yang Yu Qiao 19 80 0 29 Nov 2023
ViStruct: Visual Structural Knowledge Extraction via Curriculum Guided Code-Vision Representation Yangyi Chen Xingyao Wang Manling Li Derek Hoiem Heng Ji 30 11 0 22 Nov 2023
Examining LLMs' Uncertainty Expression Towards Questions Outside Parametric Knowledge Genglin Liu Xingyao Wang Lifan Yuan Yangyi Chen Hao Peng 29 16 0 16 Nov 2023
MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback Xingyao Wang Zihan Wang Jiateng Liu Yangyi Chen Lifan Yuan Hao Peng Heng Ji LRM 130 140 0 19 Sep 2023
Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Models Yangyi Chen Karan Sikka Michael Cogswell Heng Ji Ajay Divakaran LRM 36 24 0 08 Sep 2023