72
23

Provable Defense against Backdoor Policies in Reinforcement Learning

Abstract

We propose a provable defense mechanism against backdoor policies in reinforcement learning under subspace trigger assumption. A backdoor policy is a security threat where an adversary publishes a seemingly well-behaved policy which in fact allows hidden triggers. During deployment, the adversary can modify observed states in a particular way to trigger unexpected actions and harm the agent. We assume the agent does not have the resources to re-train a good policy. Instead, our defense mechanism sanitizes the backdoor policy by projecting observed states to a 'safe subspace', estimated from a small number of interactions with a clean (non-triggered) environment. Our sanitized policy achieves ϵ\epsilon approximate optimality in the presence of triggers, provided the number of clean interactions is O(D(1γ)4ϵ2)O\left(\frac{D}{(1-\gamma)^4 \epsilon^2}\right) where γ\gamma is the discounting factor and DD is the dimension of state space. Empirically, we show that our sanitization defense performs well on two Atari game environments.

View on arXiv
Comments on this paper

We use cookies and other tracking technologies to improve your browsing experience on our website, to show you personalized content and targeted ads, to analyze our website traffic, and to understand where our visitors are coming from. See our policy.