Personalized diffusion models have shown remarkable success in Text-to-Image (T2I) generation by enabling the injection of user-defined concepts into diverse contexts. However, balancing concept fidelity with contextual alignment remains a challenging open problem. In this work, we propose an RL-based approach that leverages the diverse outputs of T2I models to address this issue. Our method eliminates the need for human-annotated scores by generating a synthetic paired dataset for DPO-like training using external quality metrics. These better-worse pairs are specifically constructed to improve both concept fidelity and prompt adherence. Moreover, our approach supports flexible adjustment of the trade-off between image fidelity and textual alignment. Through multi-step training, our approach outperforms a naive baseline in convergence speed and output quality. We conduct extensive qualitative and quantitative analysis, demonstrating the effectiveness of our method across various architectures and fine-tuning techniques. The source code can be found atthis https URL.
View on arXiv@article{ayupov2025_2505.20975,
title={ DreamBoothDPO: Improving Personalized Generation using Direct Preference Optimization },
author={ Shamil Ayupov and Maksim Nakhodnov and Anastasia Yaschenko and Andrey Kuznetsov and Aibek Alanov },
journal={arXiv preprint arXiv:2505.20975},
year={ 2025 }
}