Machine translation (MT) has become indispensable for cross-border communication in globalized industries like e-commerce, finance, and legal services, with recent advancements in large language models (LLMs) significantly enhancing translation quality. However, applying general-purpose MT models to industrial scenarios reveals critical limitations due to domain-specific terminology, cultural nuances, and stylistic conventions absent in generic benchmarks. Existing evaluation frameworks inadequately assess performance in specialized contexts, creating a gap between academic benchmarks and real-world efficacy. To address this, we propose a three-level translation capability framework: (1) Basic Linguistic Competence, (2) Domain-Specific Proficiency, and (3) Cultural Adaptation, emphasizing the need for holistic evaluation across these dimensions. We introduce TransBench, a benchmark tailored for industrial MT, initially targeting international e-commerce with 17,000 professionally translated sentences spanning 4 main scenarios and 33 language pairs. TransBench integrates traditional metrics (BLEU, TER) with Marco-MOS, a domain-specific evaluation model, and provides guidelines for reproducible benchmark construction. Our contributions include: (1) a structured framework for industrial MT evaluation, (2) the first publicly available benchmark for e-commerce translation, (3) novel metrics probing multi-level translation quality, and (4) open-sourced evaluation tools. This work bridges the evaluation gap, enabling researchers and practitioners to systematically assess and enhance MT systems for industry-specific needs.
View on arXiv@article{li2025_2505.14244,
title={ TransBench: Benchmarking Machine Translation for Industrial-Scale Applications },
author={ Haijun Li and Tianqi Shi and Zifu Shang and Yuxuan Han and Xueyu Zhao and Hao Wang and Yu Qian and Zhiqiang Qian and Linlong Xu and Minghao Wu and Chenyang Lyu and Longyue Wang and Gongbo Tang and Weihua Luo and Zhao Xu and Kaifu Zhang },
journal={arXiv preprint arXiv:2505.14244},
year={ 2025 }
}