Inverse scaling can become U-shaped

Scaling up language models has been empirically shown to improve performance and unlock emergent abilities. Conversely, observing worse performance as a function of scale ("inverse scaling") would indicate that scaling encourages behaviors that are misaligned with human preferences. The Inverse Scaling Prize identified eleven such inverse scaling tasks, evaluated on models of up to 280B parameters and up to 500 zettaFLOPs of training compute. This paper takes a closer look at these inverse scaling tasks. We evaluate models of up to 540B parameters, trained on five times more compute than those evaluated in the Inverse Scaling Prize. With this increased range of model sizes and training compute, ten out of the eleven tasks exhibit what we call "U-shaped scaling" -- performance decreases up to a certain model size, and then increases again up to the largest model evaluated. U-shaped scaling can be seen as emergent ability unlocked by scaling and implies that inverse scaling may not hold for larger models.