Transfer among Agents: An Efficient Multiagent Transfer Learning Framework

Transfer Learning has shown great potential to enhance the single-agent Reinforcement Learning (RL) efficiency. Similarly, Multiagent RL (MARL) can also be accelerated if agents can share knowledge with each other. However, it remains a problem of how an agent should learn from other agents. In this paper, we propose a novel Multiagent Option-based Policy Transfer (MAOPT) framework to improve MARL efficiency. MAOPT learns what advice to provide and when to terminate it for each agent by modeling multiagent policy transfer as the option learning problem. Our framework provides two kinds of option learning methods in terms of what experience is used during training. One is the global option advisor, which uses the global experience for the update. The other is the local option advisor, which uses each agent's local experience when only each agent's local experiences can be obtained due to partial observability. While in this setting, each agent's experience may be inconsistent with each other, which may cause the inaccuracy and oscillation of the option-value's estimation. Therefore, we propose the successor representation option learning to solve it by decoupling the environment dynamics from rewards and learning the option-value under each agent's preference. MAOPT can be easily combined with existing deep RL and MARL approaches, and experimental results show it significantly boosts the performance of existing methods in both discrete and continuous state spaces.