178
0
v1v2 (latest)

VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan

Main:12 Pages
15 Figures
Bibliography:1 Pages
9 Tables
Appendix:12 Pages
Abstract

In this paper, we propose a comprehensive evaluation benchmark for Visual Language Models (VLM) in Traditional Chinese. Our evaluation suite, the first of its kind, contains two complementary components: (1) VisTW-MCQ, a collection of manually curated exam multi-choice questions from 21 academic subjects designed to test the broad knowledge and reasoning capabilities of VLMs; and (2) VisTW-Dialogue, an open dialogue benchmark comprising 131 image-question pairs manually created to evaluate VLMs' ability in free-form dialogue generation within Taiwanese cultural contexts. These benchmarks address a critical gap in the evaluation landscape, where existing benchmarks predominantly focus on English or Simplified Chinese, neglecting the unique linguistic and cultural aspects of Traditional Chinese used in regions like Taiwan and Hong Kong. Our analysis reveals significant performance differences across various VLMs and highlights specific challenges in processing Traditional Chinese visual content.

View on arXiv
@article{tam2025_2503.10427,
  title={ VisTW: Benchmarking Vision-Language Models for Traditional Chinese in Taiwan },
  author={ Zhi Rui Tam and Ya-Ting Pai and Yen-Wei Lee and Yun-Nung Chen },
  journal={arXiv preprint arXiv:2503.10427},
  year={ 2025 }
}
Comments on this paper

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.