Recognition of Instrument-Tissue Interactions in Endoscopic Videos via Action Triplets

Recognition of surgical activity is an essential component to developcontext-aware decision support for the operating room. In this work, we tacklethe recognition of fine-grained activities, modeled as action triplets representing the tool activity. To this end, weintroduce a new laparoscopic dataset, CholecT40, consisting of 40 videos fromthe public dataset Cholec80 in which all frames have been annotated using 128triplet classes. Furthermore, we present an approach to recognize thesetriplets directly from the video data. It relies on a module called ClassActivation Guide (CAG), which uses the instrument activation maps to guide theverb and target recognition. To model the recognition of multiple triplets inthe same frame, we also propose a trainable 3D Interaction Space, whichcaptures the associations between the triplet components. Finally, wedemonstrate the significance of these contributions via several ablationstudies and comparisons to baselines on CholecT40.