Explorations on high dimensional landscapes

The question of where does a moving particle lands on the landscape on which it is moving is a challenging one especially if the terrain is rugged. In this paper we present experimental results of such a movement in two different contexts: spherical spin glasses and two-layer networks on MNIST dataset. The unifying property in each case is that if the system is symmetric and complex enough its landscape is trivial in the sense that random initial points and a chosen descent algorithm always leads to the same level of height on the terrain. This indicates existence of energy barriers in large systems. With this observation in mind we believe modifying the model to relocate this barrier is at least as important as finding a good descent algorithm on a fixed landscape.