Evaluating the Spectral Bias of Coordinate Based MLPs

In recent years, representations given by fully connected neural networks have shown to represent scenes, objects, and other measurements well in dense low-dimensional settings. For these models, termed coordinate based MLPs, sinusoidal encodings are necessary in allowing for convergence to the high frequency components of the target function. This requirement is a result of their severe spectral bias when using dense, low dimensional coordinate based inputs. Previous work explained this phenomena using Neural Tangent Kernel (NTK) and Fourier analysis. While these methods provide insight towards this large spectral bias and the benefits of positional encoding, the properties of ReLU networks that induce this behavior are not fully determined. Analyzing spectral bias directly through the computations of ReLU networks would expose their limitations in dense settings, while providing a clearer explanation as to how this behavior emerges during the learning process. In this paper, we systematically analyze the spectral bias of a coordinate based MLP through its activation regions and gradient descent dynamics. This allows us to relate the network's expressive capacity to the speed at which gradient descent converges for components of varying frequency, and how the density of the data further restricts the model.