Masked Motion Predictors are Strong 3D Action Representation Learners

In 3D human action recognition, limited supervised data makes it challengingto fully tap into the modeling potential of powerful networks such astransformers. As a result, researchers have been actively investigatingeffective self-supervised pre-training strategies. In this work, we show thatinstead of following the prevalent pretext task to perform maskedself-component reconstruction in human joints, explicit contextual motionmodeling is key to the success of learning effective feature representation for3D action recognition. Formally, we propose the Masked Motion Prediction (MAMP)framework. To be specific, the proposed MAMP takes as input the maskedspatio-temporal skeleton sequence and predicts the corresponding temporalmotion of the masked human joints. Considering the high temporal redundancy ofthe skeleton sequence, in our MAMP, the motion information also acts as anempirical semantic richness prior that guide the masking process, promotingbetter attention to semantically rich temporal regions. Extensive experimentson NTU-60, NTU-120, and PKU-MMD datasets show that the proposed MAMPpre-training substantially improves the performance of the adopted vanillatransformer, achieving state-of-the-art results without bells and whistles. Thesource code of our MAMP is available at https://github.com/maoyunyao/MAMP.