Pseudo-LiDAR from Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving

3D object detection is an essential task in autonomous driving. Recenttechniques excel with highly accurate detection rates, provided the 3D inputdata is obtained from precise but expensive LiDAR technology. Approaches basedon cheaper monocular or stereo imagery data have, until now, resulted indrastically lower accuracies --- a gap that is commonly attributed to poorimage-based depth estimation. However, in this paper we argue that it is notthe quality of the data but its representation that accounts for the majorityof the difference. Taking the inner workings of convolutional neural networksinto consideration, we propose to convert image-based depth maps topseudo-LiDAR representations --- essentially mimicking the LiDAR signal. Withthis representation we can apply different existing LiDAR-based detectionalgorithms. On the popular KITTI benchmark, our approach achieves impressiveimprovements over the existing state-of-the-art in image-based performance ---raising the detection accuracy of objects within the 30m range from theprevious state-of-the-art of 22% to an unprecedented 74%. At the time ofsubmission our algorithm holds the highest entry on the KITTI 3D objectdetection leaderboard for stereo-image-based approaches. Our code is publiclyavailable at https://github.com/mileyan/pseudo_lidar.