Non-asymptotic Theory for the Plug-in Rule in Functional Estimation

The plug-in rule is widely used in estimating functionals of finite dimensional parameters, and involves plugging in an asymptotically efficient estimator for the parameter to obtain an asymptotically efficient estimator for the functional. We propose a general non-asymptotic theory for analyzing the performance of the plug-in rule, and demonstrate its utility by applying it to estimation of functionals of discrete distributions via the maximum likelihood estimator (MLE). We show that existing theory is insufficient for analyzing the bias of the plug-in rule, and propose to apply the theory of approximation using positive linear operators to study this bias. The variance is controlled using the well-known tools from the literature on concentration inequalities. Our techniques completely characterize the maximum $L_2$ risk incurred by the MLE in estimating the Shannon entropy $H(P) = \sum_{i = 1}^S -p_i \ln p_i$, and $F_\alpha(P) = \sum_{i = 1}^S p_i^\alpha$ up to a constant. As corollaries, for Shannon entropy estimation, we show that it is necessary and sufficient to have $n = \omega(S)$ observations for the MLE to be consistent, where $S$ represents the alphabet size. In addition, we obtain that it is necessary and sufficient to consider $n = \omega(S^{1/\alpha})$ samples for the MLE to consistently estimate $F_\alpha(P), 0<\alpha<1$. The minimax sample complexity for both problems are $\omega(S/\ln S)$ and $\omega(S^{1/\alpha}/\ln S)$, which implies that the MLE is strictly sub-optimal. When $1<\alpha<3/2$, we show that the maximum $L_2$ rate of convergence for the MLE is $n^{-2(\alpha-1)}$ for infinite alphabet size, while the minimax $L_2$ rate is $(n\ln n)^{-2(\alpha-1)}$. When $\alpha\geq 3/2$, the MLE achieves the minimax optimal $L_2$ convergence rate $n^{-1}$ regardless of the alphabet size.