UniSeg3D 3D Scene Understanding Framework

UniSeg3D is a unified 3D scene understanding framework proposed by researchers from Huazhong University of Science and Technology in 2024.A Unified Framework for 3D Scene Understanding", published in NeurIPS 2024. This framework can implement 6 different 3D point cloud segmentation tasks in the same model, including panoptic segmentation, semantic segmentation, instance segmentation, interactive segmentation, referring segmentation, and open-vocabulary segmentation.

The niSeg3D framework unifies these tasks into one model, facilitating information sharing between tasks through shared representation and processing mechanisms, thereby improving the overall understanding of 3D scenes. The framework transfers specific knowledge between different tasks by designing knowledge distillation and contrastive learning methods, thereby enhancing model performance.

In the experimental part, UniSeg3D demonstrated performance that surpasses the current state-of-the-art methods (SOTA) in three benchmarks (ScanNet20, ScanRefer and ScanNet200).