StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following

Multi-turn instruction following capability constitutes a core competency of large language models (LLMs) in real-world applications. Existing evaluation benchmarks predominantly focus on fine-grained constraint satisfaction and domain-specific capability assessment, yet overlook the crucial structural dependency between dialogue turns that distinguishes multi-turn from single-turn interactions. This structural dependency not only reflects user intent but also establishes a second dimension for instruction following evaluation beyond constraint satisfaction. To address this gap, we propose StructFlowBench, a multi-turn instruction following benchmark with structural flow modeling. The benchmark innovatively defines a structural flow framework comprising six fundamental inter-turn relationships, which not only introduces novel structural constraints for model evaluation but also serves as generation parameters for creating customized dialogue flows tailored to specific scenarios. Adopting established LLM-based automatic evaluation methodologies, we conduct systematic evaluations of 13 leading open-source and closed-source LLMs. Experimental results reveal significant deficiencies in current models' comprehension of multi-turn dialogue structures. The code is available at \url{this https URL}.

@article{li2025_2502.14494,
  title={ StructFlowBench: A Structured Flow Benchmark for Multi-turn Instruction Following },
  author={ Jinnan Li and Jinzhe Li and Yue Wang and Yi Chang and Yuan Wu },
  journal={arXiv preprint arXiv:2502.14494},
  year={ 2025 }
}