Deep Learning Is Getting Involved in the Beauty Industry. Will Internet Celebrities and Bloggers Lose Their Jobs?

In recent years, more and more beauty bloggers have emerged on the Internet. They explain beauty tips and share the results of cosmetic trials in order to accumulate fans and cooperate with businesses to sell products.

For example, Li Jiaqi, who was very popular some time ago, is known as "Lipstick Devil"A beauty blogger. He once tried all of the following products in one live broadcast: 380 typesThe lipstick color was sold out within one minute. 14,000Lipstick record.

However, many girls who love makeup should have realized that even if they bought the same lipstick as the blogger, the effect of applying it is different. When they see the color numbers that "Li Jiaqi" tries out are beautiful, fairy-like and luxurious, but when they put them on their own lips, why are they...

That's right, it is because everyone's face shape, skin color, lip shape, etc. are different that leads to "Seller Show" and "Buyer Show"result.

So the question is, how can you know which beauty product is best for you? A company called Mira gave the answer:Use deep learning.

Deep learning also loves beauty

Many people think that terms such as artificial intelligence and deep learning have nothing to do with beauty, but Mira, a startup based in Los Angeles, doesn’t think so.

The company decided to use artificial intelligence technology to help the majority of beauty-loving ladies, such as getting makeup inspiration and purchasing suitable beauty products.

After randomly chatting with dozens of beauty professionals, the Mira team learned that the biggest difficulty currently faced by female consumers in finding suitable makeup products and beauty methods is:There is no authoritative and credible voice that can provide guidance on their individual beauty needs.

In this article, we will talk about how Mira's technical team usesDeep learning and computer vision techniquesFind examples that get to the heart of this question: Find beauty influencers, images, and videos that explain specific human eye shapes and facial skin tones.

Along the way, the Mira team leveraged three simple but powerful pieces of knowledge:Geometric transformations, triplet loss functions, and transfer learning,Solve difficult beauty inference problems with minimal human input.

AI helps you choose the most suitable eye makeup

Women who love makeup know that it can be difficult to find beauty products and methods that suit their eyes - everyone's eye shape and facial skin tone are different.

Even for the same type of eye makeup (such as smoky eye), the makeup methods used are very different depending on the eye shape.

Although Birchbox and other companies have launched some useful makeup guides, the Mira team found through surveys that beauty enthusiasts generally like to listen to professional and credible advice, especially makeup advice from people with similar eye shapes to themselves. They even value these suggestions more than the opinions of beauty experts.

Using artificial intelligence technology, we can now know how to put on makeup and what cosmetics to buy based on our eye features and other unique facial features.

The first step of AI beauty: finding similarities

Let's formalize the problem: given a set of facial photos and a small number of manually labeled photos (labeled with eye color, eyelid shape, etc.), find the distance between two eyes.Visual Similarity Measures(This is what "I have seen this sister before" means in "Dream of Red Mansions"). Then useClassifierCaptures properties of human labels.

This article will first focus on how to determine the similarity between eyes, and then explain in detail how to perform the classification task.

Raw images are not very suitable for computing visual similarity or classification tasks, because many of the similarities they contain are superficial (e.g., similar makeup, but different skin tones due to strong lighting).

These have nothing to do with the actual eye structure and facial skin color of the characters. Moreover, the original images are generally in high-dimensional space, which requires a large amount of labeled training data for classification tasks.

As shown above, if you only compare the image pixels directly, the eyes of the characters are highly similar, but if you look closely, you will find that although the characters' eye shadow, lighting and line of sight are consistent, their eye color and facial skin color are different.

Mira's first tasks are:To obtain a low-dimensional and dense mathematical representation of the eye image,That is what we call 「Nesting（embeddings）.

It only captures the image qualities needed for the task (Embedding is a categorical feature represented as a continuous-valued feature. Generally, embedding refers to mapping a high-dimensional vector to a low-dimensional space.) In this way, "embedding" should ignore this information:

• Eye pose/gaze direction
• Specific lighting conditions (and powerful filters, of course)
• No matter what kind of makeup you put on your face

AI makeup step 2: Projection transformation for image normalization

We can do this through a simple preprocessing step -Projection changesRemove the superficial similarity of an entire category.

Although there are many obvious structural differences in the cropped eye photos (such as the eyes are not centered in the photo, or rotated due to head tilt, etc.), the projection changes allow us to "Twistsong"According topiece,This ensures that the same eye landmarks are at the same coordinates.

With a little helpLinear AlgebraIn principle, we can "distort" an image so that a set of points will be mapped to a new ideal shape. The process of rotating and stretching an image is as follows:

Using projection transformation, the above image can be distorted, and the 4 red dots in the above image will form a rectangle, thus "straightening" the text surrounded by the red dots. The Mira team applied the same method when normalizing the eye photos.

The researchers then used dlib to detect facial landmarks (if you are interested in dlib, you can learn more about it in the following link: http://blog.dlib.net/2014/08/real-time-face-pose-estimation.html).

Cropping the eyes in the photo and distorting them to make sure they were aligned and consistent allowed them to focus on making sure the nesting was not affected by the pose and tilt of the head.

NextImage Normalization: Detect facial landmarks, crop the eye image, and then use a projective transformation to "warp" the eye image to a standard position.

AI Beauty Step 3: Representation Learning Using Triplet Loss Function

When the “distorted” images are directly compared, they still show some superficial similarities, including gaze direction and similar makeup. Deep learning technology is the solution to this problem.

The researchers trained aConvolutional Neural Networks,Feeding it eye photos will output vectors that are more similar between eye photos of the same person than between different people.The neural network learns to output a stable and persistent representation of each individual eye in different contexts.

Of course, what we rely on here is exactly what we said before. triplet loss function,The formula is as follows:

This details that when the function places the two "nested" locations of a specific individual (anchor and positive sample) closer than the anchor and the irrelevant individual (negative sample), the model's loss and optimization objective will decrease.

When the researchers fed photos of eyes into the model, they found that the resulting “embedding” did a good job of noting when two photos had similar eye structures and facial skin tones.

The method used here is actually very similar to Google's FaceNet, which is to generate face-level image embedding by "distorting" and consistent processing of photos and applying the triplet loss function.

AI Beauty Step 4: Merge and Nest

The researchers simply tuned the generated embedding to also support human-level eye representation - extracting all the noise data for each frame.

Using the pre-trained weights of the neural network above, the researchers adopted a new loss function that puts the average values of multiple nested groups in very close positions (relative to unrelated individuals), as shown below:

Using the pre-trained weights of a previous neural network, the researchers were able to get the network to merge eye nestings together in an averaged manner, and saw the model converge quickly.Transfer learning.

Transfer learning allows the embeddings to be merged into a more holistic representation of an individual eye. Although the neural network architecture is very complex at this point, the model is able toFast convergence.

Finally, the researchers used the data set to verify the model and found that the embedding generated by the model was able toCapturing very subtle similarities between individuals,As shown below:

Just one look at you and give you perfect makeup suggestions

By obtaining a high-quality mathematical representation of a person's eye in a single photo, the researchers were able to find similarities in the structure of people's eyes, which lays the foundation for matching a person with the appropriate eye makeup style based solely on his or her eyes.

Mira's technical team said the next task is to apply severalSupervised Learning Methods(classifying eye shapes, returning eye colors, etc.), as well as some analysis methods, to build a system that can help peopleAn AI model that provides makeup suggestions.

In other words, in the future, girls will no longer have to worry about what kind of makeup is best for their eyes and skin tone, and they will no longer have to mechanically refer to standard makeup guides and beauty bloggers' test color effects. AI will recommend makeup techniques that are more suitable for you.

In this case, beauty bloggers may lose their jobs? However, Li Jiaqi no longer has to work so hard to try on colors 380 times in one live broadcast.