Residual-Conditioned Optimal Transport: Towards Structure-Preserving Unpaired and Paired Image Restoration

Deep learning-based image restoration methods generally struggle withfaithfully preserving the structures of the original image. In this work, wepropose a novel Residual-Conditioned Optimal Transport (RCOT) approach, whichmodels image restoration as an optimal transport (OT) problem for both unpairedand paired settings, introducing the transport residual as a uniquedegradation-specific cue for both the transport cost and the transport map.Specifically, we first formalize a Fourier residual-guided OT objective byincorporating the degradation-specific information of the residual into thetransport cost. We further design the transport map as a two-pass RCOT map thatcomprises a base model and a refinement process, in which the transportresidual is computed by the base model in the first pass and then encoded as adegradation-specific embedding to condition the second-pass restoration. Byduality, the RCOT problem is transformed into a minimax optimization problem,which can be solved by adversarially training neural networks. Extensiveexperiments on multiple restoration tasks show that RCOT achieves competitiveperformance in terms of both distortion measures and perceptual quality,restoring images with more faithful structures as compared withstate-of-the-art methods.