We present 6-PACK, a deep learning approach to category-level 6D object posetracking on RGB-D data. Our method tracks in real-time novel object instancesof known object categories such as bowls, laptops, and mugs. 6-PACK learns tocompactly represent an object by a handful of 3D keypoints, based on which theinterframe motion of an object instance can be estimated through keypointmatching. These keypoints are learned end-to-end without manual supervision inorder to be most effective for tracking. Our experiments show that our methodsubstantially outperforms existing methods on the NOCS category-level 6D poseestimation benchmark and supports a physical robot to perform simplevision-based closed-loop manipulation tasks. Our code and video are availableat https://sites.google.com/view/6packtracking.