A Survey on Deep Learning Technique for Video Segmentation

Video segmentation, i.e., partitioning video frames into multiple segments or objects, plays a critical role in a broad range of practical applications, from enhancing visual effects in movie, to understanding scenes in autonomous driving, to virtual background creation in video conferencing, just to name a few. Recently, due to the renaissance of connectionism in computer vision, there has been an influx of deep learning based approaches for video segmentation that have delivered compelling performance. In this survey, we comprehensively review two basic lines of research - generic object segmentation (of unknown categories) in videos and video semantic segmentation - by introducing their respective task settings, background concepts, perceived need, development history, and main challenges. We also provide a detailed overview of representative literature on both methods and datasets. Additionally, we present quantitative performance comparisons of the reviewed methods on benchmark datasets. Finally, we point out a set of unsolved open issues in this field, and suggest possible opportunities for further research. A public website is provided to continuously track recent developments in this fast advancing field: https://github.com/tfzhou/VS-Survey.