Personalized AI assistants, a hallmark of the human-like capabilities of Large Language Models (LLMs), are a challenging application that intertwines multiple problems in LLM research. Despite the growing interest in the development of personalized assistants, the lack of an open-source conversational dataset tailored for personalization remains a significant obstacle for researchers in the field. To address this research gap, we introduce HiCUPID, a new benchmark to probe and unleash the potential of LLMs to deliver personalized responses. Alongside a conversational dataset, HiCUPID provides a Llama-3.2-based automated evaluation model whose assessment closely mirrors human preferences. We release our dataset, evaluation model, and code atthis https URL.
View on arXiv@article{mok2025_2506.01262,
title={ Exploring the Potential of LLMs as Personalized Assistants: Dataset, Evaluation, and Analysis },
author={ Jisoo Mok and Ik-hwan Kim and Sangkwon Park and Sungroh Yoon },
journal={arXiv preprint arXiv:2506.01262},
year={ 2025 }
}