Usability evaluation is critical to the impact and adoption of open source software (OSS), yet traditional methods relying on human evaluators suffer from high costs and limited scalability. To address these limitations, we introduce OSS-UAgent, an automated, configurable, and interactive agent-based usability evaluation framework specifically designed for open source software. Our framework employs intelligent agents powered by large language models (LLMs) to simulate developers performing programming tasks across various experience levels (from Junior to Expert). By dynamically constructing platform-specific knowledge bases, OSS-UAgent ensures accurate and context-aware code generation. The generated code is automatically evaluated across multiple dimensions, including compliance, correctness, and readability, providing a comprehensive measure of the software's usability. Additionally, our demonstration showcases OSS-UAgent's practical application in evaluating graph analytics platforms, highlighting its effectiveness in automating usability evaluation.
View on arXiv@article{meng2025_2505.23239,
title={ OSS-UAgent: An Agent-based Usability Evaluation Framework for Open Source Software },
author={ Lingkai Meng and Yu Shao and Long Yuan and Longbin Lai and Peng Cheng and Wenyuan Yu and Wenjie Zhang and Xuemin Lin and Jingren Zhou },
journal={arXiv preprint arXiv:2505.23239},
year={ 2025 }
}