On overcoming the Curse of Dimensionality in Neural Networks

Let $H$ be a reproducing Kernel Hilbert space. For $i=1,\cdots,N$, let $x_i\in\mathbb{R}^{d}$ and $y_i\in\mathbb{R}^{m}$ comprise our dataset. Let $f^*\in H$ be the unique global minimiser of the functional \begin{equation*} J(f) = \frac{1}{2}\Vert f\Vert_{H}^{2} + \frac{1}{N}\sum_{i=1}^{N}\frac{1}{2}\vert f(x_i)-y_i\vert^{2}. \end{equation*} In this paper we show that for each $n\in\mathbb{N}$ there exists a two layer network where the first layer has $nm$ number of basis functions $\Phi_{x_{i_k},j}$ for $i_1,\cdots,i_n\in\{1,\cdots,N\}$, $j=1,\cdots,m$ and the second layer takes a weighted summation of the first layer, such that the functions $f_n$ realised by these networks satisfy \begin{equation*} \Vert f_{n}-f^*\Vert_{H}\leq O(\frac{1}{\sqrt{n}})\enspace \text{for all}\enspace n\in\mathbb{N}. \end{equation*} Thus the error rate is independent of input dimension $d$, output dimension $m$ and data size $N$.