Netflix developers have created a new type of “green screen” technology based on artificial intelligence, enabling the creation of realistic visual effects for movies and television in real-time.
The traditional “green screen” technology is used for filming actors, who are then inserted into the foreground of virtual or pre-recorded scenes. Actors are filmed against a bright green background, which can easily be isolated and removed digitally.
This process can be done automatically with sufficient accuracy, such as in weather forecasts on TV, but it can be affected by green elements in clothing, transparent or thin objects, such as strands of hair. For greater accuracy in film and TV production, camera operators manually adjust the settings, which can sometimes take hours.
In an effort to create a faster and more precise method, Netflix has developed a technique called Magenta Green Screen (MGS). Filming is done against a backdrop of bright green LEDs, while actors are illuminated with red and blue LEDs, which together create a purple glow.
Since digital cameras separately record the red, green, and blue values for each pixel, this technology results in the formation of a green channel that records only the background, while the foreground appears black. The red and blue channels are used to record only the foreground, leaving the background black. Together, they create purple and green colors.
In real-time, editors can replace the green channel, instantly and realistically placing actors in front of another scene. Even complex areas, like transparent bottles or the area around hair strands, are processed without issue.
However, this method has a drawback. Since the foreground is recorded only in red and blue, the actors appear purple. To solve this, Netflix uses AI to restore the full spectrum of colors to the foreground with photos of the actors taken under normal lighting conditions.
The AI works quickly but not in real-time yet. However, faster methods, such as averaging the red and blue channels to approximate the green channel, work effectively enough for directors to monitor during filming.
Drew Lahat from the video production company Geiger Post in California says that the industry always welcomes new methods, but this technology involves many complex stages that may not yet match the fast pace of film and TV production.
“There are a number of aspects of this technology that, from a practical standpoint, make me question whether it’s worth using,” he says. “It may work well in a fully controlled environment, but it will have to compete with other new technologies, such as virtual sets, and convince producers in real-world conditions.”