Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

24 May 2024

Furong Huang

Papers citing "Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement"

36 / 36 papers shown

Title
Anyprefer: An Agentic Framework for Preference Data Synthesis Yiyang Zhou Zekun Wang Tianle Wang Shangyu Xing Peng Xia ... Chetan Bansal Weitong Zhang Ying Wei Joey Tianyi Zhou Huaxiu Yao 61 0 0 27 Apr 2025
Platonic Grounding for Efficient Multimodal Language Models Moulik Choraria Xinbo Wu Akhil Bhimaraju Nitesh Sekhar Yue Wu Xu Zhang Prateek Singhal L. Varshney 59 0 0 27 Apr 2025
SoTA with Less: MCTS-Guided Sample Selection for Data-Efficient Visual Reasoning Self-Improvement Qing Guo Z. Yang Chao Feng Hongjin Lu Linjie Li Chung-Ching Lin Kevin Qinghong Lin Furong Huang Lijuan Wang OODD ReLM VLM LRM 69 1 0 10 Apr 2025
Towards Visual Text Grounding of Multimodal Large Language Model Ming Li Ruiyi Zhang Jian Chen Jiuxiang Gu Yufan Zhou Franck Dernoncourt Wanrong Zhu Dinesh Manocha Tong Sun 41 2 0 07 Apr 2025
AdPO: Enhancing the Adversarial Robustness of Large Vision-Language Models with Preference Optimization Chaohu Liu Tianyi Gui Yu Liu Linli Xu VLM AAML 68 1 0 02 Apr 2025
Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models Zhaochen Wang Yujun Cai Zi Huang Bryan Hooi Yiwei Wang Ming Yang CoGe VLM 73 0 0 02 Apr 2025
Aligning Multimodal LLM with Human Preference: A Survey Tao Yu Yuyao Zhang Chaoyou Fu Junkang Wu Jinda Lu ... Qingsong Wen Z. Zhang Yan Huang Liang Wang Tieniu Tan 164 2 0 18 Mar 2025
DecAlign: Hierarchical Cross-Modal Alignment for Decoupled Multimodal Representation Learning Chengxuan Qian Shuo Xing Shawn Li Yue Zhao Zhengzhong Tu 55 0 0 14 Mar 2025
DynCIM: Dynamic Curriculum for Imbalanced Multimodal Learning Chengxuan Qian Kai Han J. Wang Zhenlong Yuan Rui Qian Chongwen Lyu Jun Chen 51 1 0 09 Mar 2025
SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner Kejia Chen Jiawen Zhang Jiacong Hu Jiazhen Yang Jian Lou Zunlei Feng Mingli Song 66 0 0 06 Mar 2025
Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization Shuo Xing Yuping Wang Peiran Li Ruizheng Bai Yixuan Wang Chengxuan Qian Huaxiu Yao Zhengzhong Tu 97 6 0 18 Feb 2025
MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs Yuhang Zhou Giannis Karamanolakis Victor Soto Anna Rumshisky Mayank Kulkarni Furong Huang Wei Ai Jianhua Lu MoMe 106 0 0 03 Feb 2025
Feedback-Driven Vision-Language Alignment with Minimal Human Supervision Giorgio Giannone Ruoteng Li Qianli Feng Evgeny Perevodchikov Rui Chen Aleix M. Martinez VLM 66 0 0 08 Jan 2025
MMedPO: Aligning Medical Vision-Language Models with Clinical-Aware Multimodal Preference Optimization Kangyu Zhu Peng Xia Yun-Qing Li Hongtu Zhu Sheng Wang Huaxiu Yao 103 1 0 09 Dec 2024
VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning Xueqing Wu Yuheng Ding Bingxuan Li Pan Lu Da Yin Kai-Wei Chang Nanyun Peng LRM 105 3 0 03 Dec 2024
Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning Di Zhang Jingdi Lei Junxian Li Xunzhi Wang Y. Liu ... Steve Yang Jianbo Wu Peng Ye Wanli Ouyang Dongzhan Zhou OffRL LRM 107 6 0 27 Nov 2024
CREAM: Consistency Regularized Self-Rewarding Language Models Zekun Wang Weilei He Zhiyuan Liang Xuchao Zhang Chetan Bansal Ying Wei Weitong Zhang Huaxiu Yao ALM 101 7 0 16 Oct 2024
Unsupervised Data Validation Methods for Efficient Model Training Yurii Paniv 37 1 0 10 Oct 2024
Towards Self-Improvement of LLMs via MCTS: Leveraging Stepwise Knowledge with Curriculum Preference Learning Xiyao Wang Linfeng Song Ye Tian Dian Yu Baolin Peng Haitao Mi Furong Huang Dong Yu LRM 52 9 0 09 Oct 2024
Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks Jiayi He Hehai Lin Q. Wang Yi Ren Fung Heng Ji ReLM LRM 101 4 0 05 Oct 2024
LLaVA-Critic: Learning to Evaluate Multimodal Models Tianyi Xiong Qing Guo Dong Guo Qinghao Ye Haoqi Fan Quanquan Gu Heng Huang Chunyuan Li MLLM VLM LRM 57 37 0 03 Oct 2024
MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation? Zhaorun Chen Yichao Du Zichen Wen Yiyang Zhou Chenhang Cui ... Jiawei Zhou Zhuokai Zhao Rafael Rafailov Chelsea Finn Huaxiu Yao EGVM MLLM 58 29 0 05 Jul 2024
Improving Visual Storytelling with Multimodal Large Language Models Xiaochuan Lin Xiangyong Chen 39 0 0 02 Jul 2024
LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentation Longchao Da Tiejin Chen Lu Cheng Hua Wei 44 11 0 01 Jul 2024
Multi-Stage Balanced Distillation: Addressing Long-Tail Challenges in Sequence-Level Knowledge Distillation Yuhang Zhou Jing Zhu Paiheng Xu Xiaoyu Liu Xiyao Wang Danai Koutra Wei Ai Furong Huang 78 4 0 19 Jun 2024
CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Models Peng Xia Ze Chen Juanxi Tian Yangrui Gong Ruibo Hou ... Jimeng Sun Zongyuan Ge Gang Li James Zou Huaxiu Yao MU VLM 69 31 0 10 Jun 2024
Teaching-Assistant-in-the-Loop: Improving Knowledge Distillation from Imperfect Teacher Models in Low-Budget Scenarios Yuhang Zhou Wei Ai 37 5 0 08 Jun 2024
Self-Exploring Language Models: Active Preference Elicitation for Online Alignment Shenao Zhang Donghan Yu Hiteshi Sharma Ziyi Yang Shuohang Wang Hany Hassan Zhaoran Wang LRM 48 28 0 29 May 2024
KTO: Model Alignment as Prospect Theoretic Optimization Kawin Ethayarajh Winnie Xu Niklas Muennighoff Dan Jurafsky Douwe Kiela 176 449 0 02 Feb 2024
Self-Rewarding Language Models Weizhe Yuan Richard Yuanzhe Pang Kyunghyun Cho Xian Li Sainbayar Sukhbaatar Jing Xu Jason Weston ReLM SyDa ALM LRM 239 298 0 18 Jan 2024
RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback M. Steyvers Yuan Yao Haoye Zhang Taiwen He Yifeng Han ... Xinyue Hu Zhiyuan Liu Hai-Tao Zheng Maosong Sun Tat-Seng Chua MLLM VLM 141 177 0 01 Dec 2023
Explore Spurious Correlations at the Concept Level in Language Models for Text Classification Yuhang Zhou Paiheng Xu Xiaoyu Liu Bang An Wei Ai Furong Huang LRM 71 20 0 15 Nov 2023
mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration Qinghao Ye Haiyang Xu Jiabo Ye Mingshi Yan Anwen Hu Haowei Liu Qi Qian Ji Zhang Fei Huang Jingren Zhou MLLM VLM 126 375 0 07 Nov 2023
BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models Junnan Li Dongxu Li Silvio Savarese Steven C. H. Hoi VLM MLLM 270 4,244 0 30 Jan 2023
Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering Pan Lu Swaroop Mishra Tony Xia Liang Qiu Kai-Wei Chang Song-Chun Zhu Oyvind Tafjord Peter Clark A. Kalyan ELM ReLM LRM 211 1,106 0 20 Sep 2022
Training language models to follow instructions with human feedback Long Ouyang Jeff Wu Xu Jiang Diogo Almeida Carroll L. Wainwright ... Amanda Askell Peter Welinder Paul Christiano Jan Leike Ryan J. Lowe OSLM ALM 313 11,953 0 04 Mar 2022