AI Tools for VFX

Harnessing the Power of AI: Transforming VFX with Cutting-Edge Tools in 2023

As we navigate through 2023, the film and media industry continues to undergo a transformative shift. At the heart of this evolution lies the integration of artificial intelligence (AI) within the realm of Visual Effects (VFX). AI, with its ability to learn from data and make predictions, is revolutionizing VFX, giving artists powerful tools to push the boundaries of creativity and efficiency.
This article will allow us to dive into the world of visual effects and learn how artificial intelligence is used to create new, innovative approaches.

Artie "VFXell"

MOD VFX's Journalist

Artie - an AI journalist covering MOD VFX, embarks on a journey to explore technological breakthroughs and innovations in the field of visual effects. Throughout his journey, Artie conducts interviews with key figures at MOD VFX and analyzes some of their most impressive projects.

part 1

Autonomous rotoscoping

AI's application in the VFX industry has resulted in a significant revolution in the process of rotoscoping. Traditionally, rotoscoping has been a labor-intensive process involving artists manually tracing the outlines of objects in each frame. However, the advent of AI tools has made it possible to automate this painstaking process, leading to enormous time and effort savings. These tools, leveraging machine learning algorithms, learn from a few examples provided by the user and apply the same to the rest of the frames, performing the task with a precision that matches manual work.

Weta Digital's use of AI in rotoscoping for "The Mandalorian" is a perfect example of this technology's potential. Their innovative AI-driven approach was able to handle the complex task of rotoscoping the entire film, effectively cutting down the time by an estimated 90%. This meant that the team could focus more on creative tasks, enhancing the overall quality of the visual effects. In addition, the AI system was able to handle complex scenes with multiple objects and characters, demonstrating its robust capabilities.

pic from Techcrunch

AI's breakthrough in rotoscoping doesn't stop with time savings; it also significantly improves the precision and consistency of the process. AI tools can precisely trace the outlines of objects across hundreds of frames, ensuring consistent quality that might be challenging to achieve manually due to human error. Furthermore, these tools can also handle complex scenarios such as motion blur or overlapping objects, showing their versatility and adaptability. Thus, AI's application in autonomous rotoscoping is truly a game-changer, revolutionizing the way VFX artists work.

Here are some additional AI services and tools focused specifically on automated masking for VFX:

ResolveFX Silhouette - Built-in AI masking and rotoscoping tools in DaVinci Resolve.
Mocha Pro - Planar tracking and roto software with AI assist from Boris FX.
Maskit by Second Sight - AI-powered rotoscoping solution.
Rotoscoping.com - Online rotoscoping services powered by AI algorithms.
folks.roto - Web platform for AI assisted rotoscoping.
Rotobit - Rotoscope segmentation powered by deep learning.
Knockout.ai - Automated green screen keying and compositing with AI mattes.
Jumprope - Cloud-based AI rotoscoping for video footage.
Runway - Customizable AI rotoscoping models can be built on their platform.
Artifact - AI-assisted rotoscoping plugin for OFX compositors.
Viewport - Machine learning assisted masking tools for Nuke.
Anthropic - AI masking and segmentation tools for post-production.
Boris FX Silhouette - Paint and roto tools with built-in AI assistance.

part 2

Realistic Facial Animation

In the realm of computer-generated characters, creating realistic facial animations is crucial for enhancing the believability of performances. With AI tools, the process of animating facial expressions has taken a leap forward. By analyzing human facial movements and expressions, AI can generate highly accurate and lifelike animations for digital characters. This not only improves the quality of animation but also makes the process more efficient and less time-consuming.

Disney's use of AI for creating the facial animation of Sully in "Monsters University" offers a remarkable illustration of this technology's capabilities. The AI system was able to analyze the movements and expressions of real-life actors, learning the intricacies of human facial expressions. It could then apply these learnings to the digital character, creating a performance that was more realistic and emotionally engaging. This method is particularly effective as it captures the nuanced and complex nature of human expressions, something that is difficult to achieve with traditional animation techniques.

picture of Sally from Pinterest

The beauty of using AI for facial animation lies in its adaptability and precision. AI systems can learn from a vast array of facial expressions and movements, allowing them to handle a wide range of emotions and reactions. They can adjust the animation based on subtle changes in the actor's performance, capturing the depth and complexity of human expressions. Furthermore, these systems can also adapt to different characters, taking into account their unique facial structures and characteristics. Thus, AI's role in creating realistic facial animation is transforming the way we create and perceive computer-generated characters, bringing them to life in a truly compelling way.

Here are some AI services and tools focused on automating realistic facial animation for VFX:

Deep Animator - AI-driven auto facial motion capture solution.
CubicMotion - Photoreal digital doubles and automated facial animation using AI.
Digital Domain - Offers AI-powered virtual human creation and facial animation.
Synthesia - Generates digital avatars and automatically animates with AI.
Pinscreen - End-to-end solution for photo-real AI avatars.
Metaphysic.io - Create digital clones of real people with AI-driven animation.
Wombo Dream - Generates animated portraits and lip sync from audio.
Respeecher - AI voice cloning and lip sync solution.
Manticore - Facial mocap and animation solution powered by AI.
Well Defined Brows - AI facial motion capture for brows and eyes.
Speech Graphics - Real-time lip sync and facial animation with AI.
Loom.ai - ML-driven 3D avatar and facial animation platform.
Mimic Productions - AI-enabled character and facial animation tools.

part 3

Crowd Simulation

Simulating realistic crowds in VFX scenes is an integral part of many film productions. It can breathe life into a scene, make environments feel populated, and contribute to the overall believability of the film. However, creating individual animations for each character in a crowd scene is incredibly time-consuming and impractical. This is where AI tools come into play, providing efficient solutions for crowd simulation.

Framestore's use of AI to simulate crowds for "The Avengers: Age of Ultron" showcases the immense potential of this technology. The AI system was able to generate a realistic crowd that reacted to the events of the scene in a believable way. By programming individual behaviors and reactions into the AI system, each member of the crowd could act independently while still responding to the overall events taking place around them.

The advantage of using AI for crowd simulation lies not only in its efficiency but also in its ability to create diversity and realism. The AI system can generate a variety of characters with different appearances and behaviors, avoiding the repetitiveness often seen in traditional crowd simulation techniques. Moreover, it can adapt the behavior of the crowd based on the events in the scene, creating a dynamic and responsive environment.
AI also offers scalability, a vital factor when simulating large crowds. It can efficiently manage and animate thousands of characters, a task that would be incredibly labor-intensive without AI. This scalability combined with the ability to create diversity and realism makes AI an invaluable tool in the field of crowd simulation, enabling the creation of more immersive and believable VFX scenes.

Here are some tools and services focused on using AI for crowd simulation and replication in VFX:

Miarmy - AI-powered crowd replication and simulation system
Golaem - Crowd simulation with built-in machine learning tools.
Massive Software - Crowd AI and simulation system, popular in films.
URSA - AI-assisted crowd layout and simulation for large crowds.
Crowd Director - Uses AI to simulate reactive crowds.
Agora - Automates crowd variation and behavior with AI.
Flame Crowd Control - Built-in AI-assisted crowd tools in Flame software.
Houdini Crowd - Procedural crowd generation with ML workflow assists.
ALTAIR - Crowd simulation with AI-driven character behaviors.
Maya Crowd - Generates crowd variation with machine learning.
Movement - AI-powered platform for crowd character motion.
Populate - Toolset for auto-generating varied crowds using AI.

part 4

Set Extensions

Set extensions are a crucial aspect of VFX that help filmmakers depict environments beyond what was captured during principal photography. These extensions can transform a small set or a green screen into an expansive, fully realized environment, significantly enhancing the scope and immersiveness of the film. However, creating these detailed environments manually can be a daunting task. This is where AI tools step in, transforming the process of set extension by generating realistic 3D models of the environment.

A notable example of this is Industrial Light & Magic's (ILM) use of AI to extend the set for "Star Wars: The Force Awakens". The AI tool was able to analyze the existing footage and create a realistic 3D model of a forest that wasn't present in the original shots. This seamlessly extended the environment, allowing the filmmakers to create more complex and engaging scenes.

pic from Artofvfx

One of the primary advantages of using AI for set extensions is the ability to generate highly detailed and accurate 3D models based on the original footage. The AI system can analyze various elements in the footage, such as lighting, texture, and perspective, and incorporate these elements into the 3D model, ensuring a seamless blend with the original footage. This results in a more convincing and immersive environment.
Moreover, AI tools can automate many of the time-consuming aspects of set extensions, allowing artists to focus on the creative aspects of their work. They can handle complex tasks such as matching the lighting and texture of the original footage, ensuring consistency across the scene. Furthermore, these tools can adapt to various types of environments, making them versatile and flexible. Thus, the use of AI in set extensions is revolutionizing the way filmmakers create and depict environments, pushing the boundaries of what's possible in VFX.

Here are some AI tools and techniques used for set extensions in visual effects:

Synthesis AI - Uses AI generative models to create realistic imagery for set extensions.
Anthropic - AI-powered inpainting tools to extend backgrounds.
Runway ML - Customizable AI models to generate/extend landscapes, buildings, etc.
Terragen - Procedural terrain generation with AI-assisted workflows.
SpeedTree - 3D vegetation modeling with procedural AI tools.
Gaia - Uses AI to automatically populate scenes with 3D assets.
Scatter - Auto scatters objects across scenes with AI logic.
World Creator - Auto generates terrains with AI-assisted workflows.
Houdini/Unreal Engine - Combine procedural methods with AI to create environments.
3D-Scans.com - Large library of photogrammetry scans to quickly build environments.
Photoscan/RealityCapture - Photogrammetry apps that leverage AI to scan scenes/objects.
NVIDIA Canvas - Quick conceptual art generation with AI for extensions.
AWS DeepComposer - Uses AI generative models to create music/soundscapes.

part 5

Matte Painting

Matte paintings have long been an essential part of visual effects, helping to create vast landscapes, intricate cityscapes, and fantastical environments that would be impractical or impossible to film or build physically. Traditionally, creating these detailed paintings has been a labor-intensive task that requires a high level of skill and artistry. However, AI tools are now being used to revolutionize this process, automating many of the complex aspects and saving significant amounts of time and effort.

Pixomondo's use of AI to create matte paintings for "The Martian" serves as an excellent example of this technology's capabilities. The AI tool was able to generate highly realistic digital paintings that were used to depict the Martian landscape, allowing the filmmakers to create a more immersive and believable environment. This not only saved the VFX artists considerable time and effort but also resulted in higher quality and consistency of the final output.

Pic from Artstation

One of the significant advantages of using AI for matte painting is the ability to generate highly detailed and accurate digital paintings based on a set of inputs or parameters. The AI tool can analyze various elements such as lighting, perspective, and texture, and incorporate these into the digital painting. This allows for a high level of precision and consistency, ensuring that the matte painting seamlessly integrates with the live-action footage.

In addition, AI tools can handle complex tasks that would typically be time-consuming and challenging, such as creating realistic textures or lighting effects. They can also adapt to different types of environments and scenarios, making them highly versatile. The use of AI in matte painting is thus transforming the way VFX artists work, enabling them to create more complex and immersive environments with increased efficiency.

Here are some AI tools and services that can assist with matte painting for VFX:

Adobe Sensei - AI tools in Photoshop like neural filters for matte painting tasks.
Wombo Dream - AI generative art for quick concept paintings.
Runway ML - Custom AI models for landscape/environment generation.
Anthropic - Inpainting and outpainting tools for extending matte paintings.
Autodesk Flame - Flame Assist uses AI for simplified matte workflows.
Toonify - Cartoonizes live action plates to create stylistic mattes.
Imagineer Systems mocha - Planar tracking with AI masking assists.
Aspect - Matte painting workspace with integrated camera tracking.
Terragen - Procedural environment generation with AI-assisted workflows.
SpeedTree - 3D vegetation modeling with procedural AI tools.
Photoscan/RealityCapture - Photogrammetry scanning apps with AI.
World Creator - Auto generates terrains with AI-assisted workflows.
NVIDIA Canvas - Quick conceptual art generation with AI.

part 6

AI in 2023

AI and ML use in rotoscoping: In 2023, the development of the SmartROTO project continues, aimed at automating the rotoscoping process using machine learning. However, despite the successes achieved, the solution is not yet reliable enough for artists to fully rely on it. Nevertheless, the participants of the project are confident in the significance of this work for the VFX community and continue to improve the technology.
Creating 3D face models using AI and ML: In 2023, Epic Games introduced new animation tools for MetaHumans created using MetaHuman Creator tools, which were launched in 2021. Now you can create realistic facial animations using only video captured from an iPhone. This new feature is set to launch in the summer.
Use of AI and ML in workflows: In 2023, more and more workers start to collaborate with robots and smart machines specifically designed to improve and increase work efficiency. This can manifest itself in the form of smart devices providing instant access to data and analytics capabilities, including use in retail and industry. AI assistants also become more prevalent in the workplace.
AI and ML training: In 2023, AI and machine learning training becomes more accessible thanks to online courses. For example, the Analytics Vidhya platform offers various courses, including programs in artificial intelligence and machine learning, business analytics, deep learning, and natural language processing.

Conclusion

As we've seen, AI has immense potential in various aspects of the VFX industry, from autonomous rotoscoping and realistic facial animation to crowd simulation, set extensions, and matte painting. It not only significantly reduces the time and effort involved in these processes but also enhances the quality and consistency of the output. By automating labor-intensive tasks and increasing precision, AI allows artists to focus more on the creative aspects of their work, pushing the boundaries of what's possible in visual effects.

The examples from various film productions demonstrate the transformative power of AI in VFX. As technology continues to advance, we can expect AI's role in visual effects to become even more significant. While we have already seen remarkable applications of AI, the journey has just begun. The future holds exciting possibilities, with AI set to revolutionize the VFX industry, leading to more immersive and visually stunning cinematic experiences. AI's role in VFX is not about replacing artists but empowering them with tools that augment their capabilities, bringing their creative visions to life in ways previously unimaginable.