Model-Free Algorithm and Regret Analysis for MDPs with Peak Constraints

In the optimization of dynamic systems, the variables typically have constraints. Such problems can be modeled as a Constrained Markov Decision Process (CMDP). This paper considers the peak constraints, where the agent chooses the policy to maximize the long-term average reward as well as satisfies the constraints at each time. We propose a model-free algorithm that converts CMDP problem to an unconstrained problem and a Q-learning based approach is used. The proposed algorithm achieves $\tilde{O}(T^{\frac{1}{2}+\epsilon}\sqrt{H^4SA})$ bound for the regret and $O(HT^{\frac{1}{2}+\epsilon})$ bound for the number of constraint violations where $\epsilon>0$ is an arbitrary positive number, $T$ is the time-horizon, $S$ and $A$ is the number of states and actions, respectively, and $H$ is the number of steps per episode. We note that this is the first results on regret analysis for CMDP with peak constraints, where the transition problems are not known apriori. We demonstrate the proposed algorithm on an energy harvesting problem where it outperforms state-of-the-art and performs close to the theoretical upper bound of the studied optimization problem.