What and How Well You Performed? A Multitask Learning Approach to Action Quality Assessment

Can performance on the task of action quality assessment (AQA) be improved byexploiting a description of the action and its quality? Current AQA and skillsassessment approaches propose to learn features that serve only one task -estimating the final score. In this paper, we propose to learn spatio-temporalfeatures that explain three related tasks - fine-grained action recognition,commentary generation, and estimating the AQA score. A new multitask-AQAdataset, the largest to date, comprising of 1412 diving samples was collectedto evaluate our approach (https://github.com/ParitoshParmar/MTL-AQA). We showthat our MTL approach outperforms STL approach using two different kinds ofarchitectures: C3D-AVG and MSCADC. The C3D-AVG-MTL approach achieves the newstate-of-the-art performance with a rank correlation of 90.44%. Detailedexperiments were performed to show that MTL offers better generalization thanSTL, and representations from action recognition models are not sufficient forthe AQA task and instead should be learned.