Entropy Inference and the James-Stein Estimator

Entropy is a fundamental quantity in statistics and machine learning. In this note, we present a novel procedure for statistical learning of entropy from high-dimensional small-sample data. Specifically, we introduce a a simple yet very powerful small-sample estimator of the Shannon entropy based on James-Stein-type shrinkage. This results in an estimator that is highly efficient statistically as well as computationally. Despite its simplicity, we show that it outperforms (in part substantially) eight other competing entropy estimation procedures across a diverse range of sampling scenarios and data-generating models, including in cases of severe undersampling. A computer program is available that implements the proposed estimator.