In this paper, we propose a novel system named Disp R-CNN for 3D objectdetection from stereo images. Many recent works solve this problem by firstrecovering a point cloud with disparity estimation and then apply a 3Ddetector. The disparity map is computed for the entire image, which is costlyand fails to leverage category-specific prior. In contrast, we design aninstance disparity estimation network (iDispNet) that predicts disparity onlyfor pixels on objects of interest and learns a category-specific shape priorfor more accurate disparity estimation. To address the challenge from scarcityof disparity annotation in training, we propose to use a statistical shapemodel to generate dense disparity pseudo-ground-truth without the need of LiDARpoint clouds, which makes our system more widely applicable. Experiments on theKITTI dataset show that, even when LiDAR ground-truth is not available attraining time, Disp R-CNN achieves competitive performance and outperformsprevious state-of-the-art methods by 20% in terms of average precision.