Playing against a stationary opponent

This paper investigates properties of Blackwell $\epsilon$-optimal strategies in zero-sum stochastic games when the adversary is restricted to stationary strategies, motivated by applications to robust Markov decision processes. For a class of absorbing games, we show that Markovian Blackwell $\epsilon$-optimal strategies may fail to exist, yet we prove the existence of Blackwell $\epsilon$-optimal strategies that can be implemented by a two-state automaton whose internal transitions are independent of actions. For more general absorbing games, however, there need not exist Blackwell $\epsilon$-optimal strategies that are independent of the adversary's decisions. Our findings point to a contrast between absorbing games and generalized Big Match games, and provide new insights into the properties of optimal policies for robust Markov decision processes.

@article{grand-clément2025_2503.15346,
  title={ Playing against a stationary opponent },
  author={ Julien Grand-Clément and Nicolas Vieille },
  journal={arXiv preprint arXiv:2503.15346},
  year={ 2025 }
}