MULAN: Multitask Universal Lesion Analysis Network for Joint Lesion Detection, Tagging, and Segmentation

When reading medical images such as a computed tomography (CT) scan,radiologists generally search across the image to find lesions, characterizeand measure them, and then describe them in the radiological report. Toautomate this process, we propose a multitask universal lesion analysis network(MULAN) for joint detection, tagging, and segmentation of lesions in a varietyof body parts, which greatly extends existing work of single-task lesionanalysis on specific body parts. MULAN is based on an improved Mask R-CNNframework with three head branches and a 3D feature fusion strategy. Itachieves the state-of-the-art accuracy in the detection and tagging tasks onthe DeepLesion dataset, which contains 32K lesions in the whole body. We alsoanalyze the relationship between the three tasks and show that tag predictionscan improve detection accuracy via a score refinement layer.