Continuous Learning: Engineering Super Features With Feature Algebras

In this paper we consider a problem of searching a space of predictive models for a given training data set. We propose an iterative procedure for deriving a sequence of improving models and a corresponding sequence of sets of non-linear features on original input space. After finite number of iterations $N$ the non-linear features become $2^N$-degree polynomials on original space. We show that in a limit of infinite number of iterations derived non-linear features must form an algebra, so for any given input point a product of two features is a linear combination of features from same feature space. Due to convexity of each iteration and its ability to fall back to solutions found in previous iteration the models in the sequence have always increasing likelihood with each iteration while dimensionality of each model parameter space is set to a limited controlled value.