A key technical challenge in performing 6D object pose estimation from RGB-Dimage is to fully leverage the two complementary data sources. Prior workseither extract information from the RGB image and depth separately or usecostly post-processing steps, limiting their performances in highly clutteredscenes and real-time applications. In this work, we present DenseFusion, ageneric framework for estimating 6D pose of a set of known objects from RGB-Dimages. DenseFusion is a heterogeneous architecture that processes the two datasources individually and uses a novel dense fusion network to extractpixel-wise dense feature embedding, from which the pose is estimated.Furthermore, we integrate an end-to-end iterative pose refinement procedurethat further improves the pose estimation while achieving near real-timeinference. Our experiments show that our method outperforms state-of-the-artapproaches in two datasets, YCB-Video and LineMOD. We also deploy our proposedmethod to a real robot to grasp and manipulate objects based on the estimatedpose.