GTC '19 Classic Review | How to Choreograph and Create Dances in the Second Dimension?

At GTC China 2019 held in Suzhou recently, NetEase Leihuo Fuxi Lab introduced its AI choreography project to improve the performance of animations in games. By giving traditional manual choreography technology to AI, dance content that meets game quality can be obtained in a shorter time. Therefore, it can promote more players to interact and share, and at the same time present Song poetry and dance to the public in the form of games.

Tang Shiyi is a national first-class actor of the China National Opera and Dance Theater, and has won many major domestic dance competitions; "Against the Current", a martial arts online game created by NetEase Leihuo Fuxi Laboratory, is adapted from Wen Ruian's novel "The Four".

Who would have thought that one day, Tang Shiyi would wear a motion capture suit and appear in the production team of "Against the Cold". Her dancing posture and her charm were instantly projected onto the game character Li Shishi.

This is a part of the choreography performed by the development team of Ni Shui Han using AI. The development team cooperated with the team of China National Opera and Dance Theater, hoping to implant Chinese culture and classical dance into the game, so that the cultural treasures can be passed on in the form of game elements.

In order to improve the quality and creativity of the animation, in addition to inviting professional choreographers to choreograph dances according to the corresponding music,The team also applied artificial intelligence to the choreography process, allowing AI to learn to choreograph dances that match different music based on these professional movements.

 Leihuo Fuxi Laboratory uses AI to create a better gaming experience

NetEase Leihuo Fuxi Lab (hereinafter referred to as the Lab), a subsidiary of NetEase Group, was founded in 2007 and is NetEase's first game studio in Hangzhou. The Lab has developed three flagship game brands, "A Chinese Ghost Story", "Tianyu" and "Ni Shui Han", and has made attempts in the fields of action, sports and casual games.

In recent years, the laboratory has been continuously developing AI and has launched multiple AI-supported applications in games to create a technological gaming experience for players. For example, in "Trendy Basketball", 3v3 battle robots trained with multiple agents are used; in "Tianyu", artificial intelligence algorithms are used to detect cheats.

In the dance choreography project for the characters in "Against the Water", the laboratory also applied artificial intelligence technology in order to improve the performance of animation in the game and increase the efficiency of choreography.With the support of AI technology, players can not only set the character's dance moves by themselves, but also set the scene and lighting and give full play to their creativity.

How is AI choreography technology implemented? Let's listen to the detailed explanation from the developers behind this technology. 

Traditional choreography is inefficient, how to solve it?

At GTC China 2019 held in Suzhou on December 19 this year, Wen Xiang, a senior algorithm engineer from NetEase Leihuo Fuxi Lab, gave a speech titled《Intelligent choreography system based on video information guidance》The speech took us to explore how to choreograph with artificial intelligence.

Wen Xiang introduced that dance can easily resonate with users, soMore and more scenes will rely on dance content.Interact with players. For example, video games, VR interactions, etc. Wen Xiang and his intelligent animation team also hope toUse technology to improve the efficiency of animation production and enhance the performance of animation.

For animation dance content production,There are two traditional methods:One is that professional choreographers arrange the dance according to the music and use professional equipment to separate the dance movements, which are then modified by professional animators to get the final work. The other is that when dance content needs to be mass-produced, a dancer with dance experience is needed to cut the music and then piece the already-made dance movements into the music.

However, both methods are very time-consuming and labor-intensive.So, Wen Xiang and his team wondered if they could use AI to optimize the choreography process and automatically get the dance moves that match a piece of music. They put their ideas into practice. 

After listening to music and watching videos, AI learned choreography

The problem the team faced at the beginning was that, on the one hand, the data of music and dance movements had a lot of noise; on the other hand, the amount of data did not meet expectations. In order to solve these problems, the team found that there were many very professional choreography videos on platforms such as Douyin and YouTube, which contained music and 2D skeleton information. And they could use the existing 2D skeleton algorithm to analyze the 2D skeleton information of the music and finally match it with 3D movements. Based on this idea, the AI choreography project was launched.

 AI choreography first step: music-movement translation network 

The first step in AI choreography is to translate music and dance movements into usable data, namely the Audio-Motion Interpreter Network.

First, we selected a certain amount of professional dance videos from YouTube, including Urban, hip-hop, Chinese dance and other types. Then we captured about 200 samples, each of which was 3 to 6 minutes long. 2D skeleton detection,preprocessing the data.

The team added some ethical rules to the problems of storyboards, occlusion, and camera movement that would affect the 2D skeleton display. After processing, the music and 2D skeleton data were obtained.

Using deep learning to extract human dance postures into 2D skeleton dataAfterwards, the team usedMixture Density Network (MDN), for detailed prediction of body posture,The core of this step is to obtain 2D skeleton movement information that matches the music.

 AI Choreography Step 2: Phase-based 3D Motion Matching 

The second step is to match the 3D motion to the 2D skeleton movement information, which is called Phase-based Dance Matching.

This step is mainly based on the analysis of action orientation. The elements here includeGlobal motion features, and local curve information,For example, the curve information of the upper or lower body. Then, based on the theory that 16 beats is a cycle, the movements are divided. In the process of segmentation, it is not randomly divided, but based on a breakpoint The movement at this point is often considered to be the turning point of the dance movement.

In this way, a dance move is divided into more sub-movements. These movements are then matched with the movement information in the 3D movement library. Each set of matches will be scored, and the movement with the highest score will be the final most reasonable and beautiful output result. Then, through such matching, music is finally input to obtain a complete 3D dance move.

 The third step of AI choreography is to make dance more professional and add movement variables 

Matching the movements is not enough. The team found that in activities such as Dance Storm and Street Dance Battle, the dancing of masters always differs a little. Through observation, they found that the main reason is thatMusic CheckoutFor some links, experts will have different opinions.

Therefore, the team joinedAnimation Warping,We hope that AI matching can also do this. Therefore, when matching the closest tapping music point, they will stretch or compress the music point, and make the action curve more accelerated when approaching the tapping point, so thatGet a higher degree of fit in the stepping point.

AI choreography inspires unlimited imagination

It is reported that the project has been applied toMarketing tools, gamesIn the game "Ni Shui Han", some players even used AI choreography to make the male game characters dance enchantingly, attracting the attention of netizens.

In order to avoid producing the same dance, the team also added randomness to the output network. Therefore, the same music will produce different dances, and players do not have to worry about "wearing the same clothes" in the works.

It has to be said that with the help of AI technology, Chinese culture can be transformed into game elements, presented to the public, and spread more widely. At the same time, games are becoming more and more interesting and personalized.

To borrow the words of Li Renjie, head of NetEase's Fuxi Artificial Intelligence Laboratory, "AI makes games better and the world a better place."

-- over--