Decision Tree Algorithms for the Contextual Bandit Problem

To address the contextual bandit problem, we propose online decision tree algorithms. The analysis of proposed algorithms is based on the sample complexity needed to find the optimal decision stump. Then, the decision stumps are assembled in a decision tree, KMD-Tree, and in a random collection of decision trees, KMD-Forest. We show that the proposed algorithms are optimal up to a factor $1/\Delta$ in logarithms. The dependence of the sample complexity upon the number of contextual variables is logarithmic. The computational cost of the proposed algorithm with respect to the time horizon is linear. These analytical results allow the proposed algorithms to be efficient in real applications, where the number of events to process is huge, and where we expect that some contextual variables, chosen in a large set, have potentially non-linear dependencies with the rewards. When KMD-Tree are assembled in a KMD-Forest, the analysis is done against a strong reference, the Random Forest built knowing joint distribution of contexts and rewards. We show that the number of time steps needed to find this strong reference is logarithmic with respect to time horizon. In experiments done to illustrate the theoretical analysis, KMD-Tree and KMD-Forest obtain promising results in comparison with state-of-the-art algorithms.