ResearchTrend.AI
  • Papers
  • Communities
  • Events
  • Blog
  • Pricing
Papers
Communities
Social Events
Terms and Conditions
Pricing
Parameter LabParameter LabTwitterGitHubLinkedInBlueskyYoutube

© 2025 ResearchTrend.AI, All rights reserved.

  1. Home
  2. Papers
  3. 2003.00153
19
221

Learning Near Optimal Policies with Low Inherent Bellman Error

29 February 2020
Andrea Zanette
A. Lazaric
Mykel Kochenderfer
Emma Brunskill
    OffRL
ArXivPDFHTML
Abstract

We study the exploration problem with approximate linear action-value functions in episodic reinforcement learning under the notion of low inherent Bellman error, a condition normally employed to show convergence of approximate value iteration. First we relate this condition to other common frameworks and show that it is strictly more general than the low rank (or linear) MDP assumption of prior work. Second we provide an algorithm with a high probability regret bound O~(∑t=1HdtK+∑t=1Hdt\IBEK)\widetilde O(\sum_{t=1}^H d_t \sqrt{K} + \sum_{t=1}^H \sqrt{d_t} \IBE K)O(∑t=1H​dt​K​+∑t=1H​dt​​\IBEK) where HHH is the horizon, KKK is the number of episodes, \IBE\IBE\IBE is the value if the inherent Bellman error and dtd_tdt​ is the feature dimension at timestep ttt. In addition, we show that the result is unimprovable beyond constants and logs by showing a matching lower bound. This has two important consequences: 1) it shows that exploration is possible using only \emph{batch assumptions} with an algorithm that achieves the optimal statistical rate for the setting we consider, which is more general than prior work on low-rank MDPs 2) the lack of closedness (measured by the inherent Bellman error) is only amplified by dt\sqrt{d_t}dt​​ despite working in the online setting. Finally, the algorithm reduces to the celebrated \textsc{LinUCB} when H=1H=1H=1 but with a different choice of the exploration parameter that allows handling misspecified contextual linear bandits. While computational tractability questions remain open for the MDP setting, this enriches the class of MDPs with a linear representation for the action-value function where statistically efficient reinforcement learning is possible.

View on arXiv
Comments on this paper