Interpretable 3D Human Action Analysis with Temporal Convolutional Networks

The discriminative power of modern deep learning models for 3D human actionrecognition is growing ever so potent. In conjunction with the recentresurgence of 3D human action representation with 3D skeletons, the quality andthe pace of recent progress have been significant. However, the inner workingsof state-of-the-art learning based methods in 3D human action recognition stillremain mostly black-box. In this work, we propose to use a new class of modelsknown as Temporal Convolutional Neural Networks (TCN) for 3D human actionrecognition. Compared to popular LSTM-based Recurrent Neural Network models,given interpretable input such as 3D skeletons, TCN provides us a way toexplicitly learn readily interpretable spatio-temporal representations for 3Dhuman action recognition. We provide our strategy in re-designing the TCN withinterpretability in mind and how such characteristics of the model is leveragedto construct a powerful 3D activity recognition method. Through this work, wewish to take a step towards a spatio-temporal model that is easier tounderstand, explain and interpret. The resulting model, Res-TCN, achievesstate-of-the-art results on the largest 3D human action recognition dataset,NTU-RGBD.