Integration of the 3D Environment for On-Board UAV Visual Object Tracking

Visual object tracking from an unmanned aerial vehicle (UAV) poses several challenges such as object occlusion or background clutter. In order to improve the robustness of on-board UAV visual object tracking, we propose a pipeline combining a visual object tracker and a sparse 3D reconstruction of the static environment. The 3D reconstruction is computed with an image-based structure-from-motion (SfM) component and enables us to leverage a state estimator in the corresponding 3D scene space. This improves the handling of occlusions and artifacts caused by background clutter. We performed an evaluation on prototypical image sequences, captured from a UAV with oblique views at low altitude, by adapting an existing dataset for visual object tracking and reconstructing the observed scene in 3D. The experimental results demonstrate that the proposed approach outperforms methods using plain visual cues as well as approaches leveraging image space-based state estimations.