Detecting lesions from computed tomography (CT) scans is an important butdifficult problem because non-lesions and true lesions can appear similar. 3Dcontext is known to be helpful in this differentiation task. However, existingend-to-end detection frameworks of convolutional neural networks (CNNs) aremostly designed for 2D images. In this paper, we propose 3D context enhancedregion-based CNN (3DCE) to incorporate 3D context information efficiently byaggregating feature maps of 2D images. 3DCE is easy to train and end-to-end intraining and inference. A universal lesion detector is developed to detect allkinds of lesions in one algorithm using the DeepLesion dataset. Experimentalresults on this challenging task prove the effectiveness of 3DCE. We havereleased the code of 3DCE inhttps://github.com/rsummers11/CADLab/tree/master/lesion_detector_3DCE.