Overcoming the Curse of Dimensionality in Neural Networks

Let $A$ be a set, $V$ a real Hilbert space. Let $H$ be a real Hilbert space of functions $f:A\to V$ for which there exists $M>0$ such that for all $f\in H$, $\sup_{x\in A}\Vert f(x)\Vert_{V}\leq M \Vert f\Vert_H$. For $i=1,\cdots,n$, let $(x_i,y_i)\in A\times V$ comprise our dataset. Let $0<q<1$ and $f^*\in H$ be the unique global minimizer of the functional \begin{equation*} u(f) = \frac{q}{2}\Vert f\Vert_{H}^{2} + \frac{1-q}{2n}\sum_{i=1}^{n}\Vert f(x_i)-y_i\Vert_{V}^{2}. \end{equation*} In this paper we show that for each $k\in\mathbb{N}$ there exists a two layer network where the first layer has $k$ basis functions associated with the Hilbert space $H$ and the second layer is a weighted sum of the first layer, such that the functions $f_k$ realized by these networks satisfy \begin{equation*} \Vert f_{k}-f^*\Vert_{H}^{2} \leq \Bigl( o(1) + \frac{C}{q^2} E\bigl[ \Vert Du_{I}(f^*)\Vert_{H^{*}}^{2} \bigr] \Bigr)\frac{1}{k}. \end{equation*} Let us note that $x_i$ do not need to be in a linear space and $y_i$ are in a possibly infinite dimensional Hilbert space $V$. The error estimate is independent of the data size $n$ and in the case $V$ is finite dimensional the error estimate is also independent of the dimension of $V$.