Behavior Planning For Connected Autonomous Vehicles Using Feedback Deep Reinforcement Learning

With the development of communication technologies, connected autonomous vehicles (CAVs) can share information with each other. We propose a novel behavior planning method for CAVs to decide actions such as whether to change lane or keep lane based on the observation and shared information from neighbors, and to make sure that there exist corresponding control maneuvers such as acceleration and steering angle to guarantee the safety of each individual autonomous vehicle. We formulate this problem as a hybrid partially observable Markov decision process (HPOMDP) to consider objectives such as improving traffic flow efficiency and driving comfort and safety requirements. The discrete state transition is determined by the proposed feedback deep Q-learning algorithm using the feedback action from an underlying controller based on control barrier functions. The feedback deep Q-learning algorithm we design aims to solve the critical challenge of reinforcement learning (RL) in a physical system: guaranteeing the safety of the system while the RL is exploring the action space to increase the reward. We prove that our method renders a forward invariant safe set for the continuous state physical dynamic model of the system while the RL agent is learning. In experiments, our behavior planning method can increase traffic flow and driving comfort compared with the intelligent driving model (IDM). We also validate that our method maintains safety during the learning process.