Relevance-Guided Modeling of Object Dynamics for Reinforcement Learning

Current deep reinforcement learning approaches incorporate minimal prior knowledge about the environment, limiting computational and sample efficiency. Objects provide a succinct and causal description of the world, and several recent works have studied unsupervised object representation learning using priors and losses over static object properties like visual consistency. However, object dynamics and interaction are critical cues for objectness. In addition, extensive research has shown humans have a working memory limited to only a small number of task relevant objects. In this paper we propose a framework for reasoning about object dynamics and behavior to rapidly determine minimal and task-specific object representations. We show the need for this reasoning over object behavior and dynamics by introducing a suite of RGBD MuJoCo object collection and avoidance tasks that, while intuitive and visually simple, confound state of the art unsupervised object representation learning algorithms. We also demonstrate the potential of this framework on a number of Atari games, using our object representation and standard RL and planning algorithms to learn over 10,000x faster than standard deep RL algorithms, and faster even than human players.