"The Wandering Earth 2" Deepfake Trial, 45+ Wu Jing "made" Younger, Transformed Into a 21-year-old Boy
"The Wandering Earth 2" has grossed over 2.4 billion yuan in just 8 days, breaking 36 records and achieving 91 milestones. Among them, the 21-year-old young actor Liu Peiqiang (played by Wu Jing) and the young and handsome Tu Hengyu (played by Andy Lau) left a deep impression on the audience.
A video released on the official Weibo account of The Wandering Earth describes the complex engineering difficulty behind the face-changing special effects.
Wu Jing: Don't be scared by my young and handsome face
Source: @The Wandering Earth Official Weibo
In a question on Zhihu titled "What are the unknown production challenges behind the scenes of "The Wandering Earth 2"?", Xu Jian, visual effects supervisor of "The Wandering Earth 2" and founder of MOREVFX, admitted that the first technical difficulty is "De-aging".
Digital age change, 45+ Wu Jing transformed into a 21-year-old handsome boy
De-aging is a visual effects technology used in film and television works. It refers to the use of digital editing or computer graphics (CG) technology to make actors look younger in specific scenes.
In the movie "The Wandering Earth 2", the characters played by actors Wu Jing, Andy Lau, Wang Zhi and others all involve the rejuvenation of the characters, that is, de-aging.
The AI model was trained with 2D footage of Wu Jing when he was young to perform de-aging and complete face-changing for real-life footage.
A little extra: According to the plot of the movie, Liu Peiqiang was born in 2023.
Take Wu Jing as an example. In reality, Wu Jing was over 45 years old when filming this movie (calculated according to Baidu Encyclopedia), but Liu Peiqiang was only 21 years old in 2044 in the movie. For an actor over 45 to challenge himself to play a 21-year-old "fresh meat", he really needs some technical support.
3 major de-aging methods
In fact, the mainstream de-aging methods in the world at that time were divided into three categories.
The method consists of two main steps:
- Creating digital humans and de-aging them
Through the expression capture device worn by the actor, the actor is scanned and a database is generated to create a digital person of the target actor. The digital person is then de-aged based on stills and video footage from the actor's early career.
- Animation of digital humans based on motion capture equipment worn by actors
According to Xu Jian, visual effects director of "The Wandering Earth 2", this method has a high budget, there is a shortage of relevant talents in China, and the equipment on the body may affect the shooting of fighting scenes and the expression of actors' emotions.
Will Smith wears equipment for motion and expression capture
In the movie "The Irishman", Industrial Light & Magic developed a de-aging technology that does not require wearing physical equipment or marking on the face. The actors are surrounded by three cameras, one for normal shooting, and the other two are responsible for recording the data needed for de-aging. The necessary data is collected by shooting infrared lenses to digitally reproduce the performance.
The data recorded by the camera is combined with the light effect to de-age the character.
In the movie "Captain America 3: Civil War", the visual effects team used Robert Downey Jr.'s early film and television works and visual materials to train an AI model based on the Deepfake algorithm and complete the face-changing.
Deepfake young version of Tony Stark
According to Xu Jian, visual effects director of "The Wandering Earth 2", the visual effects team began technical experiments a year and a half before the filming of the movie began. After testing 5 or 6 face-changing methods, the visual effects team finally decided to adopt de-aging method three. Based on the Deepfake algorithm, the AI model was trained with 2D materials of the actors when they were young. After more than 5 million iterations, the real-life materials were replaced.
Review of papers on facial rejuvenation and aging
《Production-Ready Face Re-Aging for Visual Effects》
The Disney Research team proposed a Face Re-Aging Network, or FRAN for short, which can automatically reshape facial images in videos without losing their identity, thereby achieving a rejuvenation or aging effect on the target face.
Input portrait age is 35 years old, the upper and lower rows are re-aging 65 years old and 18 years old respectively
"High Resolution Face Age Editing"
This paper proposes an encoder-decoder architecture for facial age editing, which uses a simple method to achieve high-resolution face age editing and can output images with a resolution of 1024*1024.
Based on the input face (yellow frame), high-quality face images of different age groups are output
"Only a Matter of Style: Age Transformation Using a Style-Based Regression Model"
This paper proposes an image-to-image conversion method that directly encodes real facial images into the latent space of a pre-trained unconditional GAN to perform aging transformation. It can generate facial features of all ages with just one photo of the subject, accurate to every wrinkle.
Aging results generated using the SAM architecture proposed in the paper
Pixel-level face-changing technology has a long way to go and still needs industry standards
In "The Wandering Earth 2", the presentation of the young Liu Peiqiang and Tu Hengyu is based on the Deepfake algorithm. However, through examining many related papers and combining the introduction of visual effects director Xu Jian, we learned that in the later stage of film production, professional post-production personnel are still needed to perform complex improvements and repairs on the characters after the face-changing.
This is because face-changing technology still has some shortcomings in processing some details, such as stiff expressions and unnatural eye expressions. These "inconsistencies" will undoubtedly be magnified on the big movie screen, causing discomfort to the audience.
The development of de-aging and aging algorithms still has a long way to go to achieve "perfect adaptation" with the characters in film and television works, and how to monitor the abuse of Deepfake technology is also a potential risk and challenge that the industry, the technology industry and even ordinary users have to face. I am very excited to see such magnificent scene design and visual effects in domestic science fiction movies. I look forward to Little Broken Ball 3!
References: [1] https://www.zhihu.com/question/579613527 [2] https://onlinelibrary.wiley.com/doi/full/10.1111/oli.12302 [3] https://www.respeecher.com/blog/de-aging-technology-changing-hollywood-future-film-making
Content at a glance: In the Spring Festival sci-fi movie "The Wandering Earth 2", the characters played by starring Wu Jing, Andy Lau, Sha Yi and others span 14 years. The visual effects team used Deepfake de-aging technology to present the appearance of the actors at different ages.
Keywords: De-aging Deepfake The Wandering Earth
"The Wandering Earth 2" has grossed over 2.4 billion yuan in just 8 days, breaking 36 records and achieving 91 milestones. Among them, the 21-year-old young actor Liu Peiqiang (played by Wu Jing) and the young and handsome Tu Hengyu (played by Andy Lau) left a deep impression on the audience.
A video released on the official Weibo account of The Wandering Earth describes the complex engineering difficulty behind the face-changing special effects.
Wu Jing: Don't be scared by my young and handsome face
Source: @The Wandering Earth Official Weibo
In a question on Zhihu titled "What are the unknown production challenges behind the scenes of "The Wandering Earth 2"?", Xu Jian, visual effects supervisor of "The Wandering Earth 2" and founder of MOREVFX, admitted that the first technical difficulty is "De-aging".
Digital age change, 45+ Wu Jing transformed into a 21-year-old handsome boy
De-aging is a visual effects technology used in film and television works. It refers to the use of digital editing or computer graphics (CG) technology to make actors look younger in specific scenes.
In the movie "The Wandering Earth 2", the characters played by actors Wu Jing, Andy Lau, Wang Zhi and others all involve the rejuvenation of the characters, that is, de-aging.
The AI model was trained with 2D footage of Wu Jing when he was young to perform de-aging and complete face-changing for real-life footage.
A little extra: According to the plot of the movie, Liu Peiqiang was born in 2023.
Take Wu Jing as an example. In reality, Wu Jing was over 45 years old when filming this movie (calculated according to Baidu Encyclopedia), but Liu Peiqiang was only 21 years old in 2044 in the movie. For an actor over 45 to challenge himself to play a 21-year-old "fresh meat", he really needs some technical support.
3 major de-aging methods
In fact, the mainstream de-aging methods in the world at that time were divided into three categories.
The method consists of two main steps:
- Creating digital humans and de-aging them
Through the expression capture device worn by the actor, the actor is scanned and a database is generated to create a digital person of the target actor. The digital person is then de-aged based on stills and video footage from the actor's early career.
- Animation of digital humans based on motion capture equipment worn by actors
According to Xu Jian, visual effects director of "The Wandering Earth 2", this method has a high budget, there is a shortage of relevant talents in China, and the equipment on the body may affect the shooting of fighting scenes and the expression of actors' emotions.
Will Smith wears equipment for motion and expression capture
In the movie "The Irishman", Industrial Light & Magic developed a de-aging technology that does not require wearing physical equipment or marking on the face. The actors are surrounded by three cameras, one for normal shooting, and the other two are responsible for recording the data needed for de-aging. The necessary data is collected by shooting infrared lenses to digitally reproduce the performance.
The data recorded by the camera is combined with the light effect to de-age the character.
In the movie "Captain America 3: Civil War", the visual effects team used Robert Downey Jr.'s early film and television works and visual materials to train an AI model based on the Deepfake algorithm and complete the face-changing.
Deepfake young version of Tony Stark
According to Xu Jian, visual effects director of "The Wandering Earth 2", the visual effects team began technical experiments a year and a half before the filming of the movie began. After testing 5 or 6 face-changing methods, the visual effects team finally decided to adopt de-aging method three. Based on the Deepfake algorithm, the AI model was trained with 2D materials of the actors when they were young. After more than 5 million iterations, the real-life materials were replaced.
Review of papers on facial rejuvenation and aging
《Production-Ready Face Re-Aging for Visual Effects》
The Disney Research team proposed a Face Re-Aging Network, or FRAN for short, which can automatically reshape facial images in videos without losing their identity, thereby achieving a rejuvenation or aging effect on the target face.
Input portrait age is 35 years old, the upper and lower rows are re-aging 65 years old and 18 years old respectively
"High Resolution Face Age Editing"
This paper proposes an encoder-decoder architecture for facial age editing, which uses a simple method to achieve high-resolution face age editing and can output images with a resolution of 1024*1024.
Based on the input face (yellow frame), high-quality face images of different age groups are output
"Only a Matter of Style: Age Transformation Using a Style-Based Regression Model"
This paper proposes an image-to-image conversion method that directly encodes real facial images into the latent space of a pre-trained unconditional GAN to perform aging transformation. It can generate facial features of all ages with just one photo of the subject, accurate to every wrinkle.
Aging results generated using the SAM architecture proposed in the paper
Pixel-level face-changing technology has a long way to go and still needs industry standards
In "The Wandering Earth 2", the presentation of the young Liu Peiqiang and Tu Hengyu is based on the Deepfake algorithm. However, through examining many related papers and combining the introduction of visual effects director Xu Jian, we learned that in the later stage of film production, professional post-production personnel are still needed to perform complex improvements and repairs on the characters after the face-changing.
This is because face-changing technology still has some shortcomings in processing some details, such as stiff expressions and unnatural eye expressions. These "inconsistencies" will undoubtedly be magnified on the big movie screen, causing discomfort to the audience.
The development of de-aging and aging algorithms still has a long way to go to achieve "perfect adaptation" with the characters in film and television works, and how to monitor the abuse of Deepfake technology is also a potential risk and challenge that the industry, the technology industry and even ordinary users have to face. I am very excited to see such magnificent scene design and visual effects in domestic science fiction movies. I look forward to Little Broken Ball 3!
References: [1] https://www.zhihu.com/question/579613527 [2] https://onlinelibrary.wiley.com/doi/full/10.1111/oli.12302 [3] https://www.respeecher.com/blog/de-aging-technology-changing-hollywood-future-film-making