Deep learning neural network serves as a powerful tool for visual anomaly detection (AD) and fault diagnosis, attributed to its strong abstractive interpretation ability in the representation domain. The deep features from neural networks that are pretrained on the ImageNet classification task have been proved to be useful for AD based on Gaussian discriminant analysis. However, with the ever-increasing complexity of deep learning neural networks, the set of deep features becomes massive where redundancy appears to be inevitable. The redundant features increase computational cost and degrade the performance of the AD method. In this article, we discuss the deep feature selection for the AD task and show how to reduce the redundancy in the representation domain. We propose a horizontal selection (dimensional reduction) method of features with subspace decomposition and a vertical selection to identify the most effective network layer for AD and fault diagnosis. We test the proposed method on two public datasets, one for AD task and the other for fault diagnosis of bearings. We show the significance of different network layers and feature subspaces on AD tasks and prove the effectiveness of the feature selection strategy.