Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration

In this paper, we study the theoretical properties of the projected Bellman equation (PBE) and two algorithms to solve this equation: linear Q-learning and approximate value iteration (AVI). We consider two sufficient conditions for the existence of a solution to PBE : strictly negatively row dominating diagonal (SNRDD) assumption and a condition motivated by the convergence of AVI. The SNRDD assumption also ensures the convergence of linear Q-learning, and its relationship with the convergence of AVI is examined. Lastly, several interesting observations on the solution of PBE are provided when using $\epsilon$-greedy policy.

@article{lim2025_2504.10865,
  title={ Understanding the theoretical properties of projected Bellman equation, linear Q-learning, and approximate value iteration },
  author={ Han-Dong Lim and Donghwan Lee },
  journal={arXiv preprint arXiv:2504.10865},
  year={ 2025 }
}