AI Playground: Unreal Engine-based Data Ablation Tool for Deep Learning

Machine learning requires data, but acquiring and labeling real-world data ischallenging, expensive, and time-consuming. More importantly, it is nearlyimpossible to alter real data post-acquisition (e.g., change the illuminationof a room), making it very difficult to measure how specific properties of thedata affect performance. In this paper, we present AI Playground (AIP), anopen-source, Unreal Engine-based tool for generating and labeling virtual imagedata. With AIP, it is trivial to capture the same image under differentconditions (e.g., fidelity, lighting, etc.) and with different ground truths(e.g., depth or surface normal values). AIP is easily extendable and can beused with or without code. To validate our proposed tool, we generated eightdatasets of otherwise identical but varying lighting and fidelity conditions.We then trained deep neural networks to predict (1) depth values, (2) surfacenormals, or (3) object labels and assessed each network's intra- andcross-dataset performance. Among other insights, we verified that sensitivityto different settings is problem-dependent. We confirmed the findings of otherstudies that segmentation models are very sensitive to fidelity, but we alsofound that they are just as sensitive to lighting. In contrast, depth andnormal estimation models seem to be less sensitive to fidelity or lighting andmore sensitive to the structure of the image. Finally, we tested our traineddepth-estimation networks on two real-world datasets and obtained resultscomparable to training on real data alone, confirming that our virtualenvironments are realistic enough for real-world tasks.