Eat These Data Sets and Models, Learn to Dance With AI, and Make TensorFlowBoys
The deep learning algorithm GAN can be used to achieve motion tracking and migration, copying the movements of a certain character to other people. If applied to the field of dance, everyone can become a dance king.
Recently, the second season of Street Dance of China was launched, which once again ignited the dance craze among the people.
Not long after it was aired, this high-energy show received a high score of 9.6 on Douban. The dancers' wonderful performances in the competition made the audience in front of the screen shout "Too exciting!" "Amazing!" and even couldn't help shaking along with the music.
However, if you really want to jump up, the difference between reality and imagination is probablyIt’s just missing a few Show Luos.I imagine myself as:
But in reality it is like this:
For dancers, their movements are called Hiphop, Breaking, Locking, etc., while for the spectators, it is just shaking, rolling, pointing...
Maybe I'm not destined to do street dance in this life? I might as well try square dancing...
Wait! Don't give up yet.Several bigwigs from the University of California, Berkeley, have developed an AI "secret weapon" for you, which will allow you to instantly improve your dancing skills and become the next generation of dance king.
Everyone can be a dancing king
Last August, researchers at the University of California, Berkeley, published a paper titled《Everybody dance now》Thesis,makeUsing the deep learning algorithm GAN (Generative Adversarial Networks), it is possible to copy the movements of professional performers and transfer them to anyone.,Thus achieving "Do as I do".
Let's first look at the results of the copied dance and feel it:
Deepfake face-changing technology was very popular before, but now it isThe whole person can be "Deepfake"!Let’s see how this magical operation is achieved.
The paper states,Migration ActionsThe method is generally divided into the following steps:
- Given two videos, one is the action source video and the other is the target person video;
- Then an algorithm is used to detect the professional dancers’ movements from the source video and create stick figure frames of the corresponding movements;
- Next, the deep learning algorithms of the two trained generative adversarial networks (GANs) were used to create full images of the target person and generate clearer and more realistic video images for them.
The end result is,The system can map the body movements of professional dancers onto amateur dancers.In addition to imitating movements, it alsoAble to perfectly simulate human voices and facial expressions.
The Principles Behind Black Technology Revealed
The specific principle of this black technology is as follows: the action migration pipeline is divided into three parts:
1. Posture detection:
The team used existingPose detection model OpenPose (CMU open source project),Extract body, face and hand pose keypoints from the source video. The essence of this step is to encode the body pose and ignore information such as body shape.
2. Global posture standardization:
Calculate the difference between the body shape and position of the source and target person in a given frame, and transform the source pose graph into a pose graph that matches the body shape and position of the target person.
3. Infer the image of the target person from the standardized posture graph:
Using a generative adversarial network model, the model is trained to learn the mapping from the standardized pose graph to the target person image.
In developing the system, the team used the GeForce GTX 1080 Ti GPU in the NVIDIA TITAN Xp and cuDNN accelerated by PyTorch for training and inference.
In the image conversion stage, we use the adversarial training image translation developed by NVIDIA. pix2pixHD Architecture. The face residual is predicted by the global generator of pix2pixHD. They use a single 70×70 PatchGAN discriminator for faces.
During the training process, the source and target video data are collected in slightly different ways. To ensure the quality of the target video, a mobile phone camera is used to shoot real-time footage of the target subject at a speed of 120 frames per second, and each video is at least 20 minutes long.
For the source video, we only need to get the appropriate pose detection results, so you can use high-quality videos of dance performances on the Internet.
The researchers say the system's results are not perfect yet. Although the videos it produces are mostly very realistic,Sometimes, they would reveal their true colors, such as when a body part disappears.Abnormal phenomena such as "melting".
also,Since the algorithm does not encode the clothes, it is impossible to produce a video of the clothes fluttering with the movements., the target must wear tight clothes.
If you put aside these shortcomings, the technology is exciting.
With this AI tool, even if you are a novice in dancing or your limbs are stiff and uncoordinated, you can become a "dancing master" like Aaron Kwok, Show Luo, or any dancer you like. Even Jackson's moonwalk is a piece of cake for you.
However, the Berkeley team is not the only one with a dream of dancing. Google has also put a lot of thought into combining AI and dancing.
Google AI choreographs new dance moves
At the end of last year, Damien Henry, technical project manager at Google Arts and Culture, worked with British choreographer Wayne McGregor to develop aA tool that can automatically generate dance choreography in a specific style.
McGregor, who holds an honorary doctorate of science from the University of Plymouth, has always been interested in science and technology. When he reviewed his dance videos over the past 25 years, he wondered if he could use technology to keep his performances fresh. So he asked Henry how he could use technology to continuously create new dance content.
Henry got the idea from a post on a science website that used a neural network to predict the next letter based on the handwriting of the previous letter.
So, heA similar algorithm was proposed that can predict a given movement. The dancer's posture is captured through video, and then the most likely dance moves are generated and displayed on the screen in real time.show.
This algorithm also ignores people's clothing and only captures the key points of the actor's specific posture to produce a stick figure model.
When they fed it dance videos of McGregor and his dancers, the AI learned how to dance, and the dance style it generated was very similar to McGregor's.
Although artificial intelligence still has certain limitations when it comes to dance creativity, this Google AI tool cannot invent moves it has never seen before.predictThe most likely action to occur among the actions it has learned.
also,This technology can also provide mixed style dance choreography,For example, if you insert a video of Brazilian samba into McGregor's video, AI may come up with a completely new mixed dance. Henry is not worried that it will give a weird dance, because the source of learning is still input by people.of.
AI posture tracking, more than just "dance dream"
After seeing so many techniques to help you "dance", are you eager to try them out?
Dance AI can make people who are afraid or unwilling to move move more freely and easily, and experience the fun of dance and sports. But the technology behind it is not just for fun.
Support the dance AI pose estimation,There is a huge amount of energy hidden behind it, which can help us complete body movements more accurately, such as 3D fitness learning, sports posture correction, patient rehabilitation training, and even virtual fitting and photo posture correction will bring new breakthroughs.
With this kind of development, machines will understand us better and become more familiar with our physical characteristics and behaviors, thereby helping us to better understand ourselves.
Okay, let’s stop talking. I’m going to learn dancing from AI. Do you want to come with me?