In this work, we present a novel data-driven method for robust 6DoF objectpose estimation from a single RGBD image. Unlike previous methods that directlyregressing pose parameters, we tackle this challenging task with akeypoint-based approach. Specifically, we propose a deep Hough voting networkto detect 3D keypoints of objects and then estimate the 6D pose parameterswithin a least-squares fitting manner. Our method is a natural extension of2D-keypoint approaches that successfully work on RGB based 6DoF estimation. Itallows us to fully utilize the geometric constraint of rigid objects with theextra depth information and is easy for a network to learn and optimize.Extensive experiments were conducted to demonstrate the effectiveness of3D-keypoint detection in the 6D pose estimation task. Experimental results alsoshow our method outperforms the state-of-the-art methods by large margins onseveral benchmarks. Code and video are available athttps://github.com/ethnhe/PVN3D.git.