Analyzing the Fine Structure of Distributions

One aim of data mining is the identification of interesting structures in data. The basic properties of an empirical distribution, such as skewness and eventual clipping, i.e., hard limits in value ranges, need to be assessed. Of particular interest is the question of whether the data originate from one process or contain subsets related to different states of the data producing process. Data visualization tools should deliver a sensitive picture of the univariate probability density distribution (PDF) for each feature. Visualization tools for PDFs are typically kernel density estimates and include both the classical histogram as well as modern tools such as ridgeline plots, bean plots and violin plots. Conventional methods have difficulties in visualizing the PDF in the case of uniform, multimodal, and skewed distributions and distributions with clipped data if density estimation parameters remain in a default setting. As a consequence, a new visualization tool called the mirrored density plot (MD plot), which is specifically designed to discover interesting structures in continuous features, is proposed. The MD plot does not require any parameters of density estimation to be adjusted, which makes the use of this plot compelling for nonexperts. The visualization tools are evaluated in comparison to statistical tests for the typical challenges of explorative distribution analysis. The results are presented on bimodal Gaussian and skewed distributions as well, and several features include published PDFs. In an exploratory data analysis of 12 features describing the quarterly financial statements, when statistical testing becomes a demanding task, only the MD plots can identify the structure of their PDFs. Overall, the MD plot can outperform the methods mentioned above.