LiDAR-based 4D Panoptic Segmentation via Dynamic Shifting Network

With the rapid advances of autonomous driving, it becomes critical to equipits sensing system with more holistic 3D perception. However, existing worksfocus on parsing either the objects (e.g. cars and pedestrians) or scenes (e.g.trees and buildings) from the LiDAR sensor. In this work, we address the taskof LiDAR-based panoptic segmentation, which aims to parse both objects andscenes in a unified manner. As one of the first endeavors towards this newchallenging task, we propose the Dynamic Shifting Network (DS-Net), whichserves as an effective panoptic segmentation framework in the point cloudrealm. In particular, DS-Net has three appealing properties: 1) Strong backbonedesign. DS-Net adopts the cylinder convolution that is specifically designedfor LiDAR point clouds. 2) Dynamic Shifting for complex point distributions. Weobserve that commonly-used clustering algorithms are incapable of handlingcomplex autonomous driving scenes with non-uniform point cloud distributionsand varying instance sizes. Thus, we present an efficient learnable clusteringmodule, dynamic shifting, which adapts kernel functions on the fly fordifferent instances. 3) Extension to 4D prediction. Furthermore, we extendDS-Net to 4D panoptic LiDAR segmentation by the temporally unified instanceclustering on aligned LiDAR frames. To comprehensively evaluate the performanceof LiDAR-based panoptic segmentation, we construct and curate benchmarks fromtwo large-scale autonomous driving LiDAR datasets, SemanticKITTI and nuScenes.Extensive experiments demonstrate that our proposed DS-Net achieves superioraccuracies over current state-of-the-art methods in both tasks. Notably, in thesingle frame version of the task, we outperform the SOTA method by 1.8% interms of the PQ metric. In the 4D version of the task, we surpass 2nd place by5.4% in terms of the LSTQ metric.