Towards AI Safety: A Taxonomy for AI System Evaluation

The advent of advanced AI brings to the forefront the need for comprehensive safety evaluation. However, divergent practices and terminologies across different communities (i.e., AI, software engineering, and governance), combined with the complexity of AI systems and environmental affordances (e.g., access to tools), call for a holistic evaluation approach. This paper proposes a framework for comprehensive AI system evaluation comprising three components: 1) harmonised terminology to facilitate communication across disciplines involved in AI safety evaluation; 2) a taxonomy identifying essential elements for AI system evaluation; 3) a mapping between AI lifecycle, stakeholders, and requisite evaluations for accountable AI supply chain. This framework catalyses a deeper discourse on AI system evaluation beyond model-centric approaches.