Nvidia Unveils New Plugins to Enhance Digital Human Realism in Unreal Engine 5

Nvidia has unveiled its latest technology for creating AI-driven characters that mimic human appearance and behavior.

At Unreal Fest Seattle 2024, Nvidia introduced new Unreal Engine 5 on-device plugins for Nvidia Ace. These innovations simplify the development and deployment of AI-powered MetaHuman characters on Windows PCs. Ace is a comprehensive suite of digital human technologies that harness generative AI for speech, intelligence, and animation.

New Features for Developers

Developers can now utilize the Audio2Face-3D plugin for Autodesk Maya, enabling realistic facial animations that sync lips and facial movements with audio. This user-friendly interface accelerates avatar development in Maya. The plugin includes source code, allowing developers to customize and adapt it for their preferred digital content creation (DCC) tools.

Additionally, Nvidia has launched a microservice for Unreal Engine 5 that utilizes Epic's Unreal Pixel Streaming technology. This service supports the Nvidia Ace Animation Graph microservice and offers early access for the Linux operating system. The Animation Graph microservice enhances character realism and responsiveness, enabling developers to stream their MetaHuman creations across various devices.

Enhancing MetaHuman Development with Ace

Nvidia simplifies MetaHuman creation with the ACE suite. The Nvidia Ace Unreal Engine 5 sample project serves as a valuable resource for developers aiming to integrate digital humans into games and applications. This project expands the range of on-device ACE plugins, including:

- Audio2Face-3D for lip syncing and facial animation

- Nemotron-Mini 4B Instruct for response generation

- RAG for contextual information

Developers can create a comprehensive database of contextual lore for their intellectual properties, generate relevant responses with low latency, and drive seamless MetaHuman facial animations in Unreal Engine 5. Each microservice is optimized for Windows PCs, ensuring low latency and minimal memory usage.

Getting Started with Tutorials and Tools

Nvidia has released a series of tutorials for utilizing the Unreal Engine 5 plugin. Though the new plugins will be available soon, developers can begin by ensuring they have the necessary Nvidia Ace plugin and Unreal Engine sample along with a MetaHuman character.

Autodesk Maya provides game developers and technical artists with advanced animation capabilities for producing high-quality 3D graphics. With the Audio2Face-3D plugin, developers can easily create audio-driven facial animations for any character. The streamlined user interface facilitates a smooth transition to the Unreal Engine 5 environment. The customizable source code allows modifications for various DCC tools.

To begin using Maya, developers can obtain an API key or download the Audio2Face-3D NIM. Nvidia NIM is a collection of AI inference microservices that facilitate the deployment of foundational models in any cloud or data center. Ensure you have Autodesk Maya 2023, 2024, or 2025 installed. Access the Maya ACE GitHub repository for the plugin, gRPC client libraries, test assets, and sample scenes to explore and innovate with Audio2Face-3D.

As developers pursue the deployment of digital humans through cloud platforms, managing high-fidelity character streaming poses significant computing challenges. The latest Unreal Engine 5 renderer microservice in Nvidia Ace now includes support for the Nvidia Animation Graph Microservice and is available for early access on Linux.

The Animation Graph microservice allows the creation of animation state machines and blend trees, offering a flexible node-based system for animation blending, playback, and control. The new Unreal Engine 5 renderer microservice integrates with pixel streaming, which streams data from the Animation Graph microservice. This setup allows developers to host their MetaHuman characters on a cloud server and stream rendered frames and audio to any browser and edge device using Web Real-Time Communication (WebRTC).

Developers can apply for early access to download the Unreal Engine 5 renderer microservice today, enabling them to explore Nvidia Ace and start building AI-powered game characters. The Maya Ace plugin is also available for download on GitHub.