37
2

Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythia

Zhejian Zhou
Jiayu Wang
Dahua Lin
Kai Chen
Abstract

Though Large Language Models (LLMs) have shown remarkable abilities in mathematics reasoning, they are still struggling with performing numeric operations accurately, such as addition and multiplication. Numbers can be tokenized into tokens in various ways by different LLMs and affect the numeric operations performance. Currently, there are two representatives: 1) Tokenize into 11-digit, and 2) Tokenize into 131\sim 3 digit. The difference is roughly equivalent to using different numeral systems (namely base 1010 or base 10310^{3}). In light of this, we study the scaling behavior of different numeral systems in the context of transformer-based large language models. We empirically show that a base 1010 system is consistently more data-efficient than a base 10210^{2} or 10310^{3} system across training data scale, model sizes under from-scratch training settings, while different number systems have very similar fine-tuning performances. We attribute this to higher token frequencies of a base 1010 system. Additionally, we reveal extrapolation behavior patterns on addition and multiplication. We identify that base 100100 and base 10001000 systems struggle on token-level discernment and token-level operations. We also sheds light on the mechanism learnt by the models.

View on arXiv
Comments on this paper