Reference-based Restoration of Digitized Analog Videotapes

Analog magnetic tapes have been the main video data storage device forseveral decades. Videos stored on analog videotapes exhibit unique degradationpatterns caused by tape aging and reader device malfunctioning that aredifferent from those observed in film and digital video restoration tasks. Inthis work, we present a reference-based approach for the resToration ofdigitized Analog videotaPEs (TAPE). We leverage CLIP for zero-shot artifactdetection to identify the cleanest frames of each video through textual promptsdescribing different artifacts. Then, we select the clean frames most similarto the input ones and employ them as references. We design a transformer-basedSwin-UNet network that exploits both neighboring and reference frames via ourMulti-Reference Spatial Feature Fusion (MRSFF) blocks. MRSFF blocks rely oncross-attention and attention pooling to take advantage of the most usefulparts of each reference frame. To address the absence of ground truth inreal-world videos, we create a synthetic dataset of videos exhibiting artifactsthat closely resemble those commonly found in analog videotapes. Bothquantitative and qualitative experiments show the effectiveness of our approachcompared to other state-of-the-art methods. The code, the model, and thesynthetic dataset are publicly available at https://github.com/miccunifi/TAPE.