TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight

Semantic Scene Completion (SSC) aims to perform geometric completion andsemantic segmentation simultaneously. Despite the promising results achieved byexisting studies, the inherently ill-posed nature of the task presentssignificant challenges in diverse driving scenarios. This paper introducesTALoS, a novel test-time adaptation approach for SSC that excavates theinformation available in driving environments. Specifically, we focus on thatobservations made at a certain moment can serve as Ground Truth (GT) for scenecompletion at another moment. Given the characteristics of the LiDAR sensor, anobservation of an object at a certain location confirms both 1) the occupationof that location and 2) the absence of obstacles along the line of sight fromthe LiDAR to that point. TALoS utilizes these observations to obtainself-supervision about occupancy and emptiness, guiding the model to adapt tothe scene in test time. In a similar manner, we aggregate reliable SSCpredictions among multiple moments and leverage them as semantic pseudo-GT foradaptation. Further, to leverage future observations that are not accessible atthe current time, we present a dual optimization scheme using the model inwhich the update is delayed until the future observation is available.Evaluations on the SemanticKITTI validation and test sets demonstrate thatTALoS significantly improves the performance of the pre-trained SSC model. Ourcode is available at https://github.com/blue-531/TALoS.