Unexploited Information Value in Human-AI Collaboration

Humans and AIs are often paired on decision tasks with the expectation of achieving complementary performance -- where the combination of human and AI outperforms either one alone. However, how to improve performance of a human-AI team is often not clear without knowing more about what particular information and strategies each agent employs. In this paper, we propose a model based in statistical decision theory to analyze human-AI collaboration from the perspective of what information could be used to improve a human or AI decision. We demonstrate our model on a deepfake detection task to investigate seven video-level features by their unexploited value of information. We compare the human alone, AI alone and human-AI team and offer insights on how the AI assistance impacts people's usage of the information and what information that the AI exploits well might be useful for improving human decisions.

@article{guo2025_2411.10463,
  title={ Unexploited Information Value in Human-AI Collaboration },
  author={ Ziyang Guo and Yifan Wu and Jason Hartline and Jessica Hullman },
  journal={arXiv preprint arXiv:2411.10463},
  year={ 2025 }
}