Deep neural networks are being used increasingly to automate data analysisand decision making, yet their decision-making process is largely unclear andis difficult to explain to the end users. In this paper, we address the problemof Explainable AI for deep neural networks that take images as input and outputa class probability. We propose an approach called RISE that generates animportance map indicating how salient each pixel is for the model's prediction.In contrast to white-box approaches that estimate pixel importance usinggradients or other internal network state, RISE works on black-box models. Itestimates importance empirically by probing the model with randomly maskedversions of the input image and obtaining the corresponding outputs. We compareour approach to state-of-the-art importance extraction methods using both anautomatic deletion/insertion metric and a pointing metric based onhuman-annotated object segments. Extensive experiments on several benchmarkdatasets show that our approach matches or exceeds the performance of othermethods, including white-box approaches. Project page: http://cs-people.bu.edu/vpetsiuk/rise/