Progress Estimation and Phase Detection for Sequential Processes

Process modeling and understanding are fundamental for advanced human-computer interfaces and automation systems. Much recent research focused on activity recognition and little has been done on process progress detection from sensor data. We introduced a real-time, sensor-based system for modeling, recognizing and estimating the progress of a work process. We implemented a multimodal deep learning structure to extract the relevant spatio-temporal features from multimodal sensory inputs and used a novel deep regression structure for overall completeness estimation. Using process completeness estimation with a Gaussian mixture model, our system can predict the phase for sequential processes. The performance speed, calculated from completeness estimates, allows online estimation of the remaining time. To help the training of our system, we introduced a novel rectified hyperbolic tangent (rtanh) activation function and conditional loss. Our system was tested on data obtained from a medical process (trauma resuscitation) and sports events (Olympic swimming competition). Our system outperformed existing trauma-resuscitation phase detectors with over 86% phase detection accuracy, 0.67 F1-score, under 12.65% completeness estimation error, and less than 7.5 minutes time-remaining estimation error. For the Olympic swimming dataset, our system achieved 88% accuracy, 0.58 F1-score, 6.32% completeness estimation error and an average 2.9 minute time-remaining estimation error.