Towards Fundamental Limits of Multi-armed Bandits with Random Walk Feedback

In this paper, we consider a new Multi-Armed Bandit (MAB) problem where arms are nodes in an unknown and possibly changing graph, and the agent (i) initiates random walks over the graph by pulling arms, (ii) observes the random walk trajectories, and (iii) receives rewards equal to the lengths of the walks. We provide a comprehensive understanding of this problem by studying both the stochastic and the adversarial setting. In the stochastic setting, we show that this problem is not easier than a standard MAB, although additional information is available through random walk trajectories. In the adversarial setting, we show that an extension of the exponential weight algorithm can achieve a regret bound of order $\widetilde{\mathcal{O}} \left( \sqrt{ \kappa T}\right) $, where $\kappa$ is a constant that depends on the structure of the graph, instead of the number of arms.