Maximum Likelihood Estimation of Functionals of Discrete Distributions

We propose a general framework for analyzing the performance of the MLE (Maximum Likelihood Estimator) in estimating functionals of discrete distributions, under the worst-case mean squared error criterion. We show that existing theory, which was developed to accommodate a fixed alphabet and a growing number of observations, is insufficient for analyzing the bias of the MLE, and apply the theory of approximation using positive linear operators to study this bias. The variance is controlled using the well-known tools from the literature on concentration inequalities. Our techniques yield a characterization of the worst-case $L_2$ risk incurred by the MLE in estimating the Shannon entropy $H(P) = \sum_{i = 1}^S -p_i \ln p_i$, and $F_\alpha(P) = \sum_{i = 1}^S p_i^\alpha,\alpha>0$ up to a multiplicative constant, for any alphabet size $S\leq \infty$ and sample size $n$. We show that it is necessary and sufficient to have $n \gg S$ observations for the MLE to be consistent in Shannon entropy estimation. The MLE requires $n \gg S^{1/\alpha}$ samples to consistently estimate $F_\alpha(P), 0<\alpha<1$. The minimax rate-optimal estimators for both problems require $S/\ln S$ and $S^{1/\alpha}/\ln S$ samples, which implies that the MLE has a strictly sub-optimal sample complexity. When $1<\alpha<3/2$, we show that the worst-case $L_2$ rate of convergence for the MLE is $n^{-2(\alpha-1)}$ for infinite alphabet size, while the minimax $L_2$ rate is $(n\ln n)^{-2(\alpha-1)}$. When $\alpha\geq 3/2$, the MLE achieves the optimal $L_2$ convergence rate $n^{-1}$ regardless of the alphabet size. We explicitly establish an equivalence between bias analysis of plug-in estimators for general functionals under arbitrary statistical models, and the theory of approximation using positive linear operators. This equivalence is of relevance and consequence far beyond the specific problem setting in this paper.