Small-to-Large Generalization: Data Influences Models Consistently Across Scale

Choice of training data distribution greatly influences model behavior. Yet, in large-scale settings, precisely characterizing how changes in training data affects predictions is often difficult due to model training costs. Current practice is to instead extrapolate from scaled down, inexpensive-to-train proxy models. However, changes in data do not influence smaller and larger models identically. Therefore, understanding how choice of data affects large-scale models raises the question: how does training data distribution influence model behavior across compute scale? We find that small- and large-scale language model predictions (generally) do highly correlate across choice of training data. Equipped with these findings, we characterize how proxy scale affects effectiveness in two downstream proxy model applications: data attribution and dataset selection.

@article{khaddaj2025_2505.16260,
  title={ Small-to-Large Generalization: Data Influences Models Consistently Across Scale },
  author={ Alaa Khaddaj and Logan Engstrom and Aleksander Madry },
  journal={arXiv preprint arXiv:2505.16260},
  year={ 2025 }
}