Obtaining accurate labels for instance segmentation is particularlychallenging due to the complex nature of the task. Each image necessitatesmultiple annotations, encompassing not only the object class but also itsprecise spatial boundaries. These requirements elevate the likelihood of errorsand inconsistencies in both manual and automated annotation processes. Bysimulating different noise conditions, we provide a realistic scenario forassessing the robustness and generalization capabilities of instancesegmentation models in different segmentation tasks, introducing COCO-N andCityscapes-N. We also propose a benchmark for weakly annotation noise, dubbedCOCO-WAN, which utilizes foundation models and weak annotations to simulatesemi-automated annotation tools and their noisy labels. This study sheds lighton the quality of segmentation masks produced by various models and challengesthe efficacy of popular methods designed to address learning with label noise.