Enhancing the generalization capability of deep neural networks to unseendomains is crucial for safety-critical applications in the real world such asautonomous driving. To address this issue, this paper proposes a novel instanceselective whitening loss to improve the robustness of the segmentation networksfor unseen domains. Our approach disentangles the domain-specific style anddomain-invariant content encoded in higher-order statistics (i.e., featurecovariance) of the feature representations and selectively removes only thestyle information causing domain shift. As shown in Fig. 1, our method providesreasonable predictions for (a) low-illuminated, (b) rainy, and (c) unseenstructures. These types of images are not included in the training dataset,where the baseline shows a significant performance drop, contrary to ours.Being simple yet effective, our approach improves the robustness of variousbackbone networks without additional computational cost. We conduct extensiveexperiments in urban-scene segmentation and show the superiority of ourapproach to existing work. Our code is available athttps://github.com/shachoi/RobustNet.