Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task

Large language models (LLMs) are increasingly explored as substitutes for human participants in cognitive tasks, but their ability to simulate human behavioral variability remains unclear. This study examines whether LLMs can approximate individual differences in the phonemic fluency task, where participants generate words beginning with a target letter. We evaluated 34 model configurations, varying prompt specificity, sampling temperature, and model type, and compared outputs to responses from 106 human participants. While some configurations, especially Claude 3.7 Sonnet, matched human averages and lexical preferences, none reproduced the scope of human variability. LLM outputs were consistently less diverse and structurally rigid, and LLM ensembles failed to increase diversity. Network analyses further revealed fundamental differences in retrieval structure between humans and models. These results highlight key limitations in using LLMs to simulate human cognition and behavior.

@article{qiu2025_2505.16164,
  title={ Can LLMs Simulate Human Behavioral Variability? A Case Study in the Phonemic Fluency Task },
  author={ Mengyang Qiu and Zoe Brisebois and Siena Sun },
  journal={arXiv preprint arXiv:2505.16164},
  year={ 2025 }
}