Large Language Models (LLMs) have demonstrated remarkable instruction-following capabilities across various applications. However, their performance in multilingual settings remains poorly understood, as existing evaluations lack fine-grained constraint analysis. We introduce XIFBench, a comprehensive constraint-based benchmark for assessing multilingual instruction-following abilities of LLMs, featuring a novel taxonomy of five constraint categories and 465 parallel instructions across six languages spanning different resource levels. To ensure consistent cross-lingual evaluation, we develop a requirement-based protocol that leverages English requirements as semantic anchors. These requirements are then used to validate the translations across languages. Extensive experiments with various LLMs reveal notable variations in instruction-following performance across resource levels, identifying key influencing factors such as constraint categories, instruction complexity, and cultural specificity.
View on arXiv@article{li2025_2503.07539,
title={ XIFBench: Evaluating Large Language Models on Multilingual Instruction Following },
author={ Zhenyu Li and Kehai Chen and Yunfei Long and Xuefeng Bai and Yaoyin Zhang and Xuchen Wei and Juntao Li and Min Zhang },
journal={arXiv preprint arXiv:2503.07539},
year={ 2025 }
}