Existing Convolutional Neural Network (CNN) based anomaly detection and segmentation approaches are overlysensitive or not sensitive enough to noise, resulting in anomaly patterns, partially detected in the testing stage.The previous methods may also differentiate normal and abnormal images, but they cannot identify the locationof anomaly presented in test images with high accuracy. To address this issue, we propose a two-stage CNNmodel for coarse-to-fine anomaly segmentation and detection called (TASAD). In both stages of TASAD, we trainour model on a mixture of normal and abnormal training samples. The abnormal images are obtained by insert-ing pseudo-anomaly patterns that are automatically generated from anomaly source images. We use a novel andsophisticated anomaly insertion technique to generate various anomalous samples. In the first stage, we design acoarse anomaly segmentation (CAS) model that takes a whole image as an input, while in the second stage, wetrain a fine anomaly segmentation (FAS) model on image patches. FAS model improves detection and segmenta-tion performance by refining anomaly patterns partially detected by CAS model. We train our framework onMVTec dataset and compare it with state-of-the-art (SOTA) methods. The proposed architecture leads to a com-pact model size – four times smaller than the SOTA method, while exhibiting better pixel-level accuracy. TASADcan also be applied to SOTAs to further improve their anomaly detection performance. Our experiments demon-strate that when applied to the latest SOTAs, TASAD improves the average precision (AP) performance of previ-ous methods by 6.2%. For reproducibility of the results, code is provided at https://github.com/RizwanAliQau/tasad.git.