Motivated by practical machine learning applications, we revisit the outlying sequence detection problem (Li \emph{et al.}, TIT 2014) and derive fundamental limits of optimal detection when the reject option is allowed for outlying sequences. In outlying sequence detection (OSD) one is given multiple observed sequences, where most sequences are generated i.i.d. from a nominal distribution. The task is to discern the set of outlying sequences that are generated according to anomalous distributions. In OSD, the nominal and anomalous distributions are \emph{unknown}. In this paper, we consider the case where there is a reject option for the OSD, i.e., reject the samples as insufficient for reliable outlying sequence detection (cf. Bartlett \emph{et al.}, JMLR 2008). We study the tradeoff among the probabilities of misclassification error, false alarm and false reject for tests that satisfy weak conditions on the rate of decrease of these error probabilities as a function of sequence length. We propose a second-order asymptotically optimal test which provides a finite sample approximation. We first consider the case of at most one outlying sequence and then generalize our results to multiple outlying sequences where each outlying sequence can follow a different anomalous distribution.
View on arXiv