Intelligently Generate Embroidery Patterns! Wuhan Textile University Visual Computing and Digital Textile Team Released the First Multi-stitch Embroidery Generative Adversarial Network Model, Which Was Accepted by the Top Journal TVCG
Embroidery is done in the spring garden, attracting yellow orioles to perch on the willow branches. As an important representative of intangible cultural heritage, my country's embroidery art has a long history and exquisite craftsmanship. Craftsmen use different stitches and colorful silk threads to vividly display rich themes on a square of embroidered cloth. In the past, embroidery technology was complicated and the threshold was extremely high, requiring craftsmen with professional knowledge and practical experience to complete it.In recent years, convolutional neural networks (CNNs) have demonstrated powerful capabilities in tasks such as image classification, object detection, image generation, and style transfer. Researchers have also begun to explore the use of CNNs to synthesize embroidery features in images.
However, due to the complex stitches, textures and three-dimensionality of embroidery, as well as the tiny details and irregular patterns,Therefore, CNN has limitations in the application of synthetic embroidery features, such as the inability to predict different stitch types, making it difficult to effectively extract stitch features and thus unable to effectively generate coherent and natural embroidery patterns.Therefore, designers are required to manually select and adjust the stitching type and its corresponding color. This process often takes a lot of time to achieve the desired effect.
In view of this,The Visual Computing and Digital Textiles Team of the School of Computer and Artificial Intelligence of Wuhan Textile University proposed a multi-stitch embroidery generative adversarial network model MSEmbGAN. MSEmbGAN improves the accuracy of key aspects such as texture realism and color fidelity in embroidery, becoming the first generative adversarial network model based on CNN to successfully complete embroidery prediction features.
The related research is titled "MSEmbGAN: Multi-Stitch Embroidery Synthesis via Region-Aware Texture Generation".Accepted by IEEE Transactions on Visualization and Computer Graphics (TVCG),Professor Sheng Bin from the School of Computer Science and Engineering at Shanghai Jiao Tong University is the corresponding author. TVCG is a top journal in the field of computer visualization and is listed as a Class A journal by the China Computer Federation (CCF).
Research highlights:
* MSEmbGAN is the first learning-based model to successfully synthesize multi-stitch embroidery images containing a variety of stitch textures and colors
* Two collaborative sub-networks are proposed: one is a region-aware texture generation network to ensure the diversity of embroidery textures and the accuracy of stitch features; the other is a colorization network to ensure color consistency between input and output images
* Created the largest multi-needle embroidery dataset, and it is also the first embroidery dataset with detailed annotations of single-needle and multi-needle labels
Paper address:
https://csai.wtu.edu.cn/TVCG01/index.html
Dataset download address:
https://go.hyper.ai/Jmj9k
The open source project "awesome-ai4s" brings together more than 100 AI4S paper interpretations and provides massive data sets and tools:
https://github.com/hyperai/awesome-ai4s
Dataset: Contains 30K+ images, the largest embroidery dataset known so far
The researchers used professional embroidery software (Wilcom 9.0) to create more than 30,000 images, including embroidery images and corresponding content images, and all images were adjusted to a resolution of 256 × 256.The multi-stitch embroidery dataset will be open source and contributed to other researchers in this research field.
It is worth mentioning thatThe images in the multi-needle embroidery dataset are annotated with 4 labels:There are 3 single stitch types and 1 multiple stitch type (Multiple Stitch, which refers to the mixture of 3 single stitch types). The 3 single stitch types are Satin Stitch, Tatami Stitch and Flat Stitch.
This is the first embroidery dataset that is detailedly annotated with single-stitch and multi-stitch labels. It includes more than 13,000 aligned embroidery images and more than 17,000 unaligned images. It is the largest embroidery dataset known to date.
The steps to create a multi-stitch embroidery dataset are as follows:
* Draw content images: Before making an embroidery plate, the embroiderer must draw a content image containing embroidery color information as a template. Most content images have simple colors and clear shapes, which can speed up network connection.
* Stitch design: For content images of different shapes, stitches must be selected to fill each area. The embroidery designer will match the appropriate stitch type according to the shape of each area. In addition, the relevant parameters of each stitch (such as spacing and direction) must be set reasonably for subsequent embroidery rendering tasks.
* Create embroidery dataset: Embroidery designers use professional embroidery software (Wilcom 9.0) to design and create embroidery patterns and render the corresponding embroidery images.
Model architecture: Contains two sub-networks: region-aware texture generation network and coloring network
The MSEmbGAN model first identifies the stitch types within the input image region, generates the corresponding embroidery texture based on the identified stitch types, and finally optimizes the overall color of the result.
In order to achieve the above functions, the researchers proposed two sub-networks:That is, the Region-aware Texture Generation Network (see the orange box in the figure below) and the Colorization Network (see the yellow box in the figure below).
The region-aware texture generation network consists of a stitch classifier module (Stitch Classifier, see the blue box in the figure above) and a stitch latent code generator module (Stitch Latent Code Generator, see the green box in the figure above).The region-aware texture generation network detects multiple color regions of the input image C and generates a grayscale single-needle embroidery image based on the shape characteristics of each local color region. The coloring network subnetwork further refines the overall image to ensure that the color of the generated multi-knitted fabric image is consistent with the color of the input image.
Due to the complexity of the region-aware texture generation network, the researchers trained it in two steps. The first step is to generate embroidery textures, using a reconstruction network to retain as many original image features as possible; the second step is to reconstruct color information, using a prior Gaussian distribution to generate embroidery images without a dataset.
Research results: MSEmbGAN outperforms current state-of-the-art embroidery synthesis and style transfer methods
To evaluate the performance of the MSEmbGAN model, the researchers conducted four evaluations: quantitative and qualitative, user feedback surveys, and ablation experiments.
Quantitative evaluation
In the quantitative evaluation, the researchers compared style transfer methods such as Pix2Pix, CycleGAN, MUNIT, and DRIT++ based on the constructed multi-needle embroidery dataset.As shown in the table below, the researchers quantified the comparison results and calculated the Learned Perceptual Image Patch Similarity (LPIPS) and Fréchet Inception Distance (FID).
The results show that compared with other methods, MSEmbGAN has a lower LPIPS distance, which means that the embroidery images generated by MSEmbGAN are perceptually closer to real embroidery images.In addition, the researchers used FID to measure the feature distribution of the generated embroidery images and real images, and evaluated the FID scores.The results show that the embroidery images generated by MSEmbGAN are closest to the ground truth.
Qualitative Assessment
In a qualitative evaluation, the researchers used a region-aware texture generation network to maintain the authenticity and color fidelity of the embroidery texture, making the results generated by MSEmbGAN have highly diverse embroidery textures.The results show that MSEmbGAN outperforms existing methods in both texture and color, i.e., the texture generated using MSEmbGAN is closer to the real embroidery texture, and the color is closer to the texture of the input image.
User feedback survey
To obtain subjective feedback from users, the researchers prepared 14 images, each of which was processed using the MSEmbGAN model and four other methods, and invited 25 candidates to give each generated image a score of 1-5 based on the following criteria:
Embroidery quality: whether the generated image has embroidery-related features and vivid textures
* Color quality: the color similarity between the input image and the generated image
* Image quality: the degree of texture distortion, color shift, high-frequency noise, and other artifacts
The researchers collected 5,250 ratings and calculated the mean and standard error for each criterion.A higher score means the generated embroidery image has better quality.The details are shown in the following table.
The results show thatMSEmbGAN outperforms all three criteria, and its overall performance is more stable than other methods.
Ablation experiment
In addition, the researchers conducted two ablation experiments: first to verify the role of the stitch classifier and stitch latent code generator, and then to verify the role of the coloring network and color consistency constraints.
As shown in the figure below, (a) represents the input image; (b) represents the embroidery image generated by removing the stitch classifier C(reg) and the hidden code generator G(slc); (c) represents the embroidery image generated by removing the coloring network (CN) and the color consistency constraint (CC); (d) represents the embroidery image generated using the complete MSEmbGAN.
The ablation results showed thatWithout the stitch classifier and stitch hidden code generator, the embroidery image texture style synthesized by the network is single and does not retain the multi-stitch style features. Secondly, a texture generation process is unstable and abnormal.
Similarly, if the coloring network and color consistency constraints are removed, the embroidery results synthesized by MSEmbGAN cannot maintain the color characteristics, resulting in obvious color shift. That is, there is a huge difference in the color distribution between the generated image and the input image.
Adhere to the deep integration of computer technology and textile and garment industry, and achieve remarkable results in many fields
The Visual Computing and Digital Textiles Team of the School of Computer and Artificial Intelligence of Wuhan Textile University has long been committed to research in the fields of computer vision, virtual reality, multimodal learning and intelligent computing.We insist on combining computer-related technologies with the textile and garment industry, and have achieved a series of results in smart wearables, smart fashion design and recommendation, fabric digital twins and intelligent computing, and virtual fitting.More than 100 academic papers have been published in high-level journals such as TVCG, IOT, TCE, KBS, WWW and international conferences recommended by CCF. Some of the research results of the team in recent years are as follows:
In view of the fact that existing virtual fitting methods fail to consider the relationship between the human body and clothing, resulting in distortion of clothing texture, the team proposed a highly realistic 3D virtual fitting network H3DVT+.The network establishes a global relationship between people and clothing, can deform clothing into a spatial distribution in a natural fitting state, more accurately infer the prior information of the clothing's 3D shape, and create a detailed 3D human body model.
Paper address:
https://ieeexplore.ieee.org/document/9716786
https://ieeexplore.ieee.org/document/10609455
In the research on the existing intelligent clothing sensing human physiological signals, the team proposed a method for detecting human breathing signals around the clock based on flexible sensing equipment.The extracted respiratory signals are used for real-time detection of asthma, providing theoretical support for the application of smart healthcare.
Paper address:
https://ieeexplore.ieee.org/abstract/document/10040599
at the same time,The college team built a smart clothing system composed of multiple sensors.Map the human body status information with the human body 3D model in real time.Realize the synchronous display of the human body status in the real world and the human body model status in the virtual three-dimensional space.
Paper address:
https://ieeexplore.ieee.org/document/9964338/
The team has been cooperating with high-level universities and research institutions at home and abroad. Specifically,The Visual Computing and Digital Textiles team has long cooperated with Professor Sheng Bin's team at Shanghai Jiao Tong University on multiple projects in the areas of smart textiles and big health, and has published more than 10 high-level papers.In the past five years, Professor Sheng has published 69 SCI papers as (co-)first/corresponding author in Nature Medicine, Nature Communications, Science Bulletin, IJCV, IEEE TPAMI, etc.
also,The team has carried out in-depth cooperation with universities and research institutions such as the Hong Kong Polytechnic University, the University of Wollongong in Australia, the Singapore Science and Technology Agency, and the Renmin University of China in the fields of natural language processing, intelligent fashion recommendations, multimodal learning, and large models.