LSTM-based early recognition of motion patterns

In this paper a method for Early Recognition (ER) of Motion Templates (MTs) is presented. We define ER as an algorithm to provide recognition results before a motion sequence is completed. In our experiments we apply Long Short-Term Memory (LSTM) and optimize the training for the task of recognizing the motion template as early as possible. The evaluation has shown that the recognition accuracy for a frame-by-frame classification the LSTM achieves a recognition accuracy of 88% if no training data of the person him/herself is included, and 92% if the training data also contains motion sequences of the person. Furthermore, the average earliness - the number of time frames it takes before the LSTM correctly classifies a motion pattern - is around 24.77 frames, which is less than a second with the used tracking technology, i.e., the Microsoft Kinect.