Shadow removal is still a challenging task due to its inherentbackground-dependent and spatial-variant properties, leading to unknown anddiverse shadow patterns. Even powerful state-of-the-art deep neural networkscould hardly recover traceless shadow-removed background. This paper proposes anew solution for this task by formulating it as an exposure fusion problem toaddress the challenges. Intuitively, we can first estimate multipleover-exposure images w.r.t. the input image to let the shadow regions in theseimages have the same color with shadow-free areas in the input image. Then, wefuse the original input with the over-exposure images to generate the finalshadow-free counterpart. Nevertheless, the spatial-variant property of theshadow requires the fusion to be sufficiently `smart', that is, it shouldautomatically select proper over-exposure pixels from different images to makethe final output natural. To address this challenge, we propose theshadow-aware FusionNet that takes the shadow image as input to generate fusionweight maps across all the over-exposure images. Moreover, we propose theboundary-aware RefineNet to eliminate the remaining shadow trace further. Weconduct extensive experiments on the ISTD, ISTD+, and SRD datasets to validateour method's effectiveness and show better performance in shadow regions andcomparable performance in non-shadow regions over the state-of-the-art methods.We release the model and code inhttps://github.com/tsingqguo/exposure-fusion-shadow-removal.