NVIDIA AI tools enable deep learning-based performance capture for creators at all levels: VFX and animation studios, creative professionals, and even any camera-equipped enthusiast .
With NVIDIA Vid2Vid Cameo, creators can leverage AI to capture their movements and facial expressions from any standard 2D video taken with a professional camera or smartphone. Performance can be applied in real time to animate an avatar, character, or painting.
And with 3D body pose estimation software, creators can capture full-body movements like walking, dancing, and martial arts, bringing virtual characters to life with AI.
For people with no 3D experience, these tools make it easy to animate creative projects, even using smartphone footage. Professionals can go further by combining pose estimation and Vid2Vid Cameo software to transfer their own movements to virtual characters for live streams or animation projects.
And creative studios can leverage AI-based performance capture for design or preview, to quickly get an idea of how certain movements look on a digital character.
NVIDIA demonstrates performance capture with Vid2Vid Cameo
NVIDIA Vid2Vid Cameo, available via a demo on the NVIDIA AI Playground, only needs two elements to generate a talking video: a still image of the avatar or painting to be animated, and footage of the artist original speaking, singing or moving his head.
Based on Generative Adversarial Networks, or GANs, the model maps facial movements to capture movement in real time, transferring that movement to the virtual character. Trained on 180,000 videos, the network learned to identify 20 key points to model facial movement – encoding the location of eyes, mouth, nose, eyebrows and more.
These points are taken from the performer’s video stream and applied to the avatar or digital character. See how it works in the demo below, which transfers a performance of Edgar Allan Poe’s “Sonnet – to Science” to a portrait of the writer by artist Gary Kelley.
Visual rigs integrate Vid2Vid Cameo, Pose Estimation by NVIDIA
While Vid2Vid Cameo captures detailed facial expressions, pose estimation AI tracks full-body movements – a key capability for creators working with virtual characters that perform complex movements or move through a digital scene. .
Pose Tracker is a convolutional neural network model available as an extension in the NVIDIA Omniverse 3D design global simulation and collaboration platform. It allows users to upload footage or stream live video as a motion source to animate a character in real time. Creators can download NVIDIA Omniverse for free and get started with step-by-step tutorials.
Companies that have integrated NVIDIA AI for performance capture into their products include:
- DerivativeMaker of TouchDesigner, a real-time, node-based visual development platform, implemented Vid2Vid Cameo as a way to provide easy-to-use facial tracking.
- Notch, a company providing a real-time graphics tool for 3D, visual effects and live event visuals, uses NVIDIA’s Body Pose Estimation AI to help artists simplify scene setups . Instead of relying on custom hardware tracking systems, Notch users can work with off-the-shelf camera gear to control 3D character animation in real time.
- Pixotope, a leading virtual production company, uses NVIDIA’s real-time AI-powered talent tracking to drive interactive elements for live productions. The Norway-based company shared its work enabling interaction between real and virtual on-screen elements at the latest NVIDIA GTC.
See NVIDIA’s latest advances in AI, digital humans, and virtual worlds at SIGGRAPH, the world’s largest gathering of computer graphics experts, running through Thursday, August 11.
