Industrial visual detection is an essential part in modern industry for equipment maintenance and inspection. With the recent progress of deep learning, advanced industrial object detectors are built for smart industrial applications. However, deep learning methods are known data-hungry: the processes of data collection and annotation are labor-intensive and time-consuming. It is especially impractical in industrial scenarios to collect publicly available datasets due to the inherent diversity and privacy. In this paper, we explore automation of industrial visual inspection and propose a segmentation-aggregation framework to learn object detectors from weakly annotated visual data. The used minimum annotation is only image-level category labels without bounding boxes. The method is implemented and evaluated on collected insulator images and public PASCAL VOC benchmarks to verify its effectiveness. The experiments show that our models achieve high detection accuracy and can be applied in industry to achieve automatic visual inspection with minimum annotation cost.