A spring-block theory of feature learning in deep neural networks

Feature-learning deep nets progressively collapse data to a regular low-dimensional geometry. How this emerges from the collective action of nonlinearity, noise, learning rate, and other factors, has eluded first-principles theories built from microscopic neuronal dynamics. We exhibit a noise-nonlinearity phase diagram that identifies regimes where shallow or deep layers learn more effectively and propose a macroscopic mechanical theory that reproduces the diagram and links feature learning across layers to generalization.

@article{shi2025_2407.19353,
  title={ A spring-block theory of feature learning in deep neural networks },
  author={ Cheng Shi and Liming Pan and Ivan Dokmanić },
  journal={arXiv preprint arXiv:2407.19353},
  year={ 2025 }
}