Collaborative problem solving (CPS) is widely recognized as a critical 21st century skill. Efficiently coding communication data is a big challenge in scaling up research on assessing CPS. This paper reports the findings on using ChatGPT to directly code CPS chat data by benchmarking performance across multiple datasets and coding frameworks. We found that ChatGPT-based coding outperformed human coding in tasks where the discussions were characterized by colloquial languages but fell short in tasks where the discussions dealt with specialized scientific terminology and contexts. The findings offer practical guidelines for researchers to develop strategies for efficient and scalable analysis of communication data from CPS tasks.
View on arXiv@article{hao2025_2411.10246,
title={ Automated Coding of Communications in Collaborative Problem-solving Tasks Using ChatGPT },
author={ Jiangang Hao and Wenju Cui and Patrick Kyllonen and Emily Kerzabi and Lei Liu and Michael Flor },
journal={arXiv preprint arXiv:2411.10246},
year={ 2025 }
}