Improved Approximation Properties of Dictionaries and Applications to Neural Networks

This article addresses the problem of approximating a function in a Hilbert space by an expansion over a dictionary $\mathbb{D}$. We introduce the notion of a smoothly parameterized dictionary and give upper bounds on the approximation rates, metric entropy and $n$-widths of the absolute convex hull, which we denote $B_1(\mathbb{D})$, of such dictionaries. The upper bounds depend upon the order of smoothness of the parameterization, and improve upon existing results in many cases. The main applications of these results is to the dictionaries $\mathbb{D} = \{\sigma(\omega\cdot x + b)\}\subset L^2$ corresponding to shallow neural networks with activation function $\sigma$, and to the dictionary of decaying Fourier modes corresponding to the spectral Barron space. This improves upon existing approximation rates for shallow neural networks when $\sigma = \text{ReLU}^k$ for $k\geq 2$, sharpens bounds on the metric entropy, and provides the first bounds on the Gelfand $n$-widths of the Barron space and spectral Barron space.