Shadow removal from a single image is generally still an open problem. Mostexisting learning-based methods use supervised learning and require a largenumber of paired images (shadow and corresponding non-shadow images) fortraining. A recent unsupervised method, Mask-ShadowGAN~\cite{Hu19}, addressesthis limitation. However, it requires a binary mask to represent shadowregions, making it inapplicable to soft shadows. To address the problem, inthis paper, we propose an unsupervised domain-classifier guided shadow removalnetwork, DC-ShadowNet. Specifically, we propose to integrate ashadow/shadow-free domain classifier into a generator and its discriminator,enabling them to focus on shadow regions. To train our network, we introducenovel losses based on physics-based shadow-free chromaticity, shadow-robustperceptual features, and boundary smoothness. Moreover, we show that ourunsupervised network can be used for test-time training that further improvesthe results. Our experiments show that all these novel components allow ourmethod to handle soft shadows, and also to perform better on hard shadows bothquantitatively and qualitatively than the existing state-of-the-art shadowremoval methods. Our code is available at:\url{https://github.com/jinyeying/DC-ShadowNet-Hard-and-Soft-Shadow-Removal}.