Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets

In addition to generating data and annotations, devising sensible datasplitting strategies and evaluation metrics is essential for the creation of abenchmark dataset. This practice ensures consensus on the usage of the data,homogeneous assessment, and uniform comparison of research methods on thedataset. This study focuses on CholecT50, which is a 50 video surgical datasetthat formalizes surgical activities as triplets of .In this paper, we introduce the standard splits for the CholecT50 and CholecT45datasets and show how they compare with existing use of the dataset. CholecT45is the first public release of 45 videos of CholecT50 dataset. We also developa metrics library, ivtmetrics, for model evaluation on surgical triplets.Furthermore, we conduct a benchmark study by reproducing baseline methods inthe most predominantly used deep learning frameworks (PyTorch and TensorFlow)to evaluate them using the proposed data splits and metrics and release thempublicly to support future research. The proposed data splits and evaluationmetrics will enable global tracking of research progress on the dataset andfacilitate optimal model selection for further deployment.