Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages

Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pretrained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.

@article{higashiyama2025_2505.22273,
  title={ Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages },
  author={ Shohei Higashiyama and Masao Utiyama },
  journal={arXiv preprint arXiv:2505.22273},
  year={ 2025 }
}