Improved Deep Reinforcement Learning with Expert Demonstrations for Urban Autonomous Driving

Currently, urban autonomous driving remains challenging because of the complexity of the driving environment. Learning-based approaches, such as reinforcement learning (RL) and imitation learning (IL), have indicated superiority over rule-based approaches, showing great potential to make decisions intelligently, but they still do not work well in urban driving situations. To better tackle this problem, this paper proposes a novel learning-based method that combines deep reinforcement learning with expert demonstrations, focusing on longitudinal motion control in autonomous driving. Our proposed method employs the soft actor-critic structure and modifies the learning process of the policy network to incorporate both the goals of maximizing reward and imitating the expert. Moreover, an adaptive prioritized experience replay is designed to sample experience from both the agent's self-exploration and expert demonstration, in order to improve the sample efficiency. The proposed method is validated in a simulated urban roundabout scenario and compared with various prevailing RL and IL baseline approaches. The results manifest that the proposed method has a faster training speed, as well as better performance in navigating safely and time-efficiently. The ablation study reveals that the prioritized replay and expert demonstration filter play important roles in our proposed method.