TruFor: Leveraging all-round clues for trustworthy image forgery detection and localization

In this paper we present TruFor, a forensic framework that can be applied toa large variety of image manipulation methods, from classic cheapfakes to morerecent manipulations based on deep learning. We rely on the extraction of bothhigh-level and low-level traces through a transformer-based fusion architecturethat combines the RGB image and a learned noise-sensitive fingerprint. Thelatter learns to embed the artifacts related to the camera internal andexternal processing by training only on real data in a self-supervised manner.Forgeries are detected as deviations from the expected regular pattern thatcharacterizes each pristine image. Looking for anomalies makes the approachable to robustly detect a variety of local manipulations, ensuringgeneralization. In addition to a pixel-level localization map and a whole-imageintegrity score, our approach outputs a reliability map that highlights areaswhere localization predictions may be error-prone. This is particularlyimportant in forensic applications in order to reduce false alarms and allowfor a large scale analysis. Extensive experiments on several datasets show thatour method is able to reliably detect and localize both cheapfakes anddeepfakes manipulations outperforming state-of-the-art works. Code is publiclyavailable at https://grip-unina.github.io/TruFor/