Spectral Analysis and Fixed Point Stability of Deep Neural Dynamics

In this work, we analyze the eigenvalue spectra and stability of discrete-time dynamical systems parametrized by deep neural networks. In particular, we leverage a representation of deep neural networks as pointwise affine maps, thus exposing their local linear operators and making them accessible to classical system analytic methods.The view of neural networks as affine parameter varying maps allows us to "crack open the black box" of neural network dynamical behavior by visualizing stationary points, state-space partitioning, and eigenvalue spectra. We provide sufficient conditions for the fixed-point stability of discrete-time deep neural dynamical systems. Empirically, we analyze the variance in dynamical behavior and eigenvalue spectra of local linear operators of neural dynamics with varying weight factorizations, activation functions, bias terms, and depths.