Concept discovery in neural networks often targets individual neurons or human-interpretable features, overlooking distributed layer-wide patterns. We study the Neural Activation Pattern (NAP) methodology, which clusters full-layer activation distributions to identify such layer-level concepts. Applied to visual object recognition and radio receiver models, we propose improved normalization, distribution estimation, distance metrics, and varied cluster selection. In the radio receiver model, distinct concepts did not emerge; instead, a continuous activation manifold shaped by Signal-to-Noise Ratio (SNR) was observed -- highlighting SNR as a key learned factor, consistent with classical receiver behavior and supporting physical plausibility. Our enhancements to NAP improved in-distribution vs. out-of-distribution separation, suggesting better generalization and indirectly validating clustering quality. These results underscore the importance of clustering design and activation manifolds in interpreting and troubleshooting neural network behavior.
View on arXiv@article{tuononen2025_2505.15570,
title={ Refining Neural Activation Patterns for Layer-Level Concept Discovery in Neural Network-Based Receivers },
author={ Marko Tuononen and Duy Vu and Dani Korpi and Vesa Starck and Ville Hautamäki },
journal={arXiv preprint arXiv:2505.15570},
year={ 2025 }
}