PEg TRAnsfer Workflow recognition challenge report: Does multi-modal data improve recognition?

This paper presents the design and results of the "PEg TRAnsfert Workflowrecognition" (PETRAW) challenge whose objective was to develop surgicalworkflow recognition methods based on one or several modalities, among video,kinematic, and segmentation data, in order to study their added value. ThePETRAW challenge provided a data set of 150 peg transfer sequences performed ona virtual simulator. This data set was composed of videos, kinematics, semanticsegmentation, and workflow annotations which described the sequences at threedifferent granularity levels: phase, step, and activity. Five tasks wereproposed to the participants: three of them were related to the recognition ofall granularities with one of the available modalities, while the othersaddressed the recognition with a combination of modalities. Averageapplication-dependent balanced accuracy (AD-Accuracy) was used as evaluationmetric to take unbalanced classes into account and because it is moreclinically relevant than a frame-by-frame score. Seven teams participated in atleast one task and four of them in all tasks. Best results are obtained withthe use of the video and the kinematics data with an AD-Accuracy between 93%and 90% for the four teams who participated in all tasks. The improvementbetween video/kinematic-based methods and the uni-modality ones was significantfor all of the teams. However, the difference in testing execution time betweenthe video/kinematic-based and the kinematic-based methods has to be taken intoconsideration. Is it relevant to spend 20 to 200 times more computing time forless than 3% of improvement? The PETRAW data set is publicly available atwww.synapse.org/PETRAW to encourage further research in surgical workflowrecognition.