HighRes-net: Recursive Fusion for Multi-Frame Super-Resolution of Satellite Imagery

Generative deep learning has sparked a new wave of Super-Resolution (SR)algorithms that enhance single images with impressive aesthetic results, albeitwith imaginary details. Multi-frame Super-Resolution (MFSR) offers a moregrounded approach to the ill-posed problem, by conditioning on multiplelow-resolution views. This is important for satellite monitoring of humanimpact on the planet -- from deforestation, to human rights violations -- thatdepend on reliable imagery. To this end, we present HighRes-net, the first deeplearning approach to MFSR that learns its sub-tasks in an end-to-end fashion:(i) co-registration, (ii) fusion, (iii) up-sampling, and (iv)registration-at-the-loss. Co-registration of low-resolution views is learnedimplicitly through a reference-frame channel, with no explicit registrationmechanism. We learn a global fusion operator that is applied recursively on anarbitrary number of low-resolution pairs. We introduce a registered loss, bylearning to align the SR output to a ground-truth through ShiftNet. We showthat by learning deep representations of multiple views, we can super-resolvelow-resolution signals and enhance Earth Observation data at scale. Ourapproach recently topped the European Space Agency's MFSR competition onreal-world satellite imagery.