Object detection is a matured technique, converging to the detection performance of human vision. This paper presents a method to further close the remaining gap of detection capability by investigating visual factors impairing the detectability of objects. As some of these factors are hard or impossible to measure in real sensor data, a detector is trained on synthetic data making perfectmeasurements and ground truth data available at a large scale. The resulting detector is then used to calibrate an empirical weightingloss, which weights samples of real training data and their corresponding detection impairing factors. The method is applied to the task of pedestrian detection in traffic scenes. The effectiveness of the empirical detection impairment weighting loss (DIW loss)is demonstrated on a detector trained on the CityPersons dataset and reaches a new state-of-the-art detection performance on thisbenchmark, improving the previous by 1.88%.