We propose a novel method for detecting horizontal vanishing points and thezenith vanishing point in man-made environments. The dominant trend in existingmethods is to first find candidate vanishing points, then remove outliers byenforcing mutual orthogonality. Our method reverses this process: we propose aset of horizon line candidates and score each based on the vanishing points itcontains. A key element of our approach is the use of global image context,extracted with a deep convolutional network, to constrain the set of candidatesunder consideration. Our method does not make a Manhattan-world assumption andcan operate effectively on scenes with only a single horizontal vanishingpoint. We evaluate our approach on three benchmark datasets and achievestate-of-the-art performance on each. In addition, our approach issignificantly faster than the previous best method.