The Calibrated Kolmogorov-Smirnov Test

Continuous goodness-of-fit (GOF) is a classical hypothesis testing problem in statistics. Despite numerous suggestions, the Kolmogorov-Smirnov (KS) test is, by far, the most popular GOF test used in practice. Unfortunately, it lacks power at the tails, which is important in many practical applications. In this paper we study the tail-sensitive Calibrated KS (CKS) test statistic, which is intimately related to the works of Berk and Jones and to the Higher Criticism statistic. In contrast to KS, which considers the largest deviation between the empirical and assumed distributions, CKS looks for the deviation which is most statistically significant. We make two main contributions. First, we study some statistical properties of CKS, in particular proving its asymptotic optimality for a wide range of rare-weak mixture models. Second, we derive a novel computationally efficient method to calculate $p$-values for a broad family of one-sided GOF tests, including the one-sided version of CKS, as well as the Higher Criticism and the Berk-Jones tests. We conclude with some simulation results comparing the power of CKS to several other tests.