Learning to Abstain from Binary Prediction

We address how to learn a binary classifier capable of abstaining from making a label prediction. Such a classifier hopes to abstain where it would be most inaccurate if forced to predict, so it has two goals in tension with each other: minimizing errors, and avoiding abstaining unnecessarily often. In this work, we exactly characterize the best achievable tradeoff between these two goals in a general semi-supervised setting, given an ensemble of classifiers of varying competence as well as unlabeled data on which we wish to predict or abstain. We give an algorithm for learning a classifier which trades off its errors with abstentions in a minimax optimal manner. This algorithm is as efficient as linear learning and prediction, and comes with strong and robust theoretical guarantees. Our analysis extends to a large class of loss functions and other scenarios, including ensembles comprised of "specialist" classifiers that can themselves abstain.