We propose a 3D object detection method for autonomous driving by fullyexploiting the sparse and dense, semantic and geometry information in stereoimagery. Our method, called Stereo R-CNN, extends Faster R-CNN for stereoinputs to simultaneously detect and associate object in left and right images.We add extra branches after stereo Region Proposal Network (RPN) to predictsparse keypoints, viewpoints, and object dimensions, which are combined with 2Dleft-right boxes to calculate a coarse 3D object bounding box. We then recoverthe accurate 3D bounding box by a region-based photometric alignment using leftand right RoIs. Our method does not require depth input and 3D positionsupervision, however, outperforms all existing fully supervised image-basedmethods. Experiments on the challenging KITTI dataset show that our methodoutperforms the state-of-the-art stereo-based method by around 30% AP on both3D detection and 3D localization tasks. Code has been released athttps://github.com/HKUST-Aerial-Robotics/Stereo-RCNN.