VTGAN: Semi-supervised Retinal Image Synthesis and Disease Prediction using Vision Transformers

In Fluorescein Angiography (FA), an exogenous dye is injected in thebloodstream to image the vascular structure of the retina. The injected dye cancause adverse reactions such as nausea, vomiting, anaphylactic shock, and evendeath. In contrast, color fundus imaging is a non-invasive technique used forphotographing the retina but does not have sufficient fidelity for capturingits vascular structure. The only non-invasive method for capturing retinalvasculature is optical coherence tomography-angiography (OCTA). However, OCTAequipment is quite expensive, and stable imaging is limited to small areas onthe retina. In this paper, we propose a novel conditional generativeadversarial network (GAN) capable of simultaneously synthesizing FA images fromfundus photographs while predicting retinal degeneration. The proposed systemhas the benefit of addressing the problem of imaging retinal vasculature in anon-invasive manner as well as predicting the existence of retinalabnormalities. We use a semi-supervised approach to train our GAN usingmultiple weighted losses on different modalities of data. Our experimentsvalidate that the proposed architecture exceeds recent state-of-the-artgenerative networks for fundus-to-angiography synthesis. Moreover, our visiontransformer-based discriminators generalize quite well on out-of-distributiondata sets for retinal disease prediction.