Recent deep learning methods have achieved promising results in image shadowremoval. However, most of the existing approaches focus on working locallywithin shadow and non-shadow regions, resulting in severe artifacts around theshadow boundaries as well as inconsistent illumination between shadow andnon-shadow regions. It is still challenging for the deep shadow removal modelto exploit the global contextual correlation between shadow and non-shadowregions. In this work, we first propose a Retinex-based shadow model, fromwhich we derive a novel transformer-based network, dubbed ShandowFormer, toexploit non-shadow regions to help shadow region restoration. A multi-scalechannel attention framework is employed to hierarchically capture the globalinformation. Based on that, we propose a Shadow-Interaction Module (SIM) withShadow-Interaction Attention (SIA) in the bottleneck stage to effectively modelthe context correlation between shadow and non-shadow regions. We conductextensive experiments on three popular public datasets, including ISTD, ISTD+,and SRD, to evaluate the proposed method. Our method achieves state-of-the-artperformance by using up to 150X fewer model parameters.