Building the Metaverse using NVIDIA Omniverse

What is a Metaverse?

A Metaverse is an online virtual reality (VR) space where users can interact with a digital environment and other users. The Metaverse blends reality and digital worlds with VR technology that offers an immersive experience.

Traditionally, there are three types of Metaverse:

• Industrial/research (e.g. NVIDIA Earth1)
• Gaming (e.g. Roblox)
• Consumer (e.g. Meta)

NVIDIA Omniverse provides tools that span all three areas, allowing artists and developers to blend these areas and open up further possibilities for their creations.

What is NVIDIA Omniverse?

NVIDIA Omniverse is a scalable multi-GPU real-time graphics platform used to build and operate Multiverse applications. With C++ and Python-based development, teams can work collaboratively and seamlessly on complex projects. The platform is modular, easily extensible and can be used by both creators and developers. It provides many opportunities, from connecting 3D worlds and workflows to simulating large-scale worlds.

Getting started with NVIDIA Omniverse

NVIDIA provides two pricing plans: the Creative and Developer plans allow individuals to use the platform for free. The Enterprise plan is a paid plan that allows collaboration across teams.

Depending on whether you are a creator or developer, NVIDIA provides extensive resources to help get you started with the platform.

System requirements

NVIDIA Omniverse runs on both Windows and Linux, and requires an RTX GPU.

Of note is that you don’t necessarily need a super-powerful machine, as we observed in a demo with SmartCow2, where they used a normal developer desktop with RTX 3060.

How do you develop on the NVIDIA Omniverse platform?

With NVIDIA Omniverse, we can simulate something in the real world before putting it into production. For example, if you wanted to place a camera, you could test the location and camera angle beforehand. You can also change the dynamics to see what the camera would pick up in various lighting conditions at different times or weather conditions.

Synthetic data generation

When creating a Machine Learning model, we’ll refer to a dataset, which forms the basis of how good the model will be. Real-world data, which is collected manually, can present challenges, such as omitted values, duplicate data, bad labels, or bad feature values. With synthetic data, we can create a higher-quality dataset in an automated way.

Out-of-the-box features

Omniverse provides many pre-built models and has an extensive assets catalog, which you can use out-of-the-box. An advantage of the asset catalog is that you can get started without needing to be a 3D artist.

When generating a synthetic data set, 3D transformations (e.g. translation, rotation and scaling) are the most important elements to handle and can be automated.

If you want to create variables (e.g. different angles, weather or lighting), the Omniverse Universal Replicator tool can pick up descriptions of parameters and generate multiple interpretations of each variable. Omniverse can do this automatically, or you can code the variables yourself.

Adding and integrating features

You can add features or integrate with Omniverse using Connectors or Extensions.

Omniverse runs with many different Connectors, depending on what you want to do. With Omniverse Connect, you can connect with Apps such as Unreal Engine, Maya or Revit to make 3D creations in Omniverse.

Extensions can help to expand the functionality of the Apps being used. For example, if you want to create a digital twin of a human, there are extensions which allow you to start with a human skeleton and then paint on top of it. The platform can show you thousands of versions and how behavior changes in different situations. There are also options to sync audio with video.

Creators can also build or customize their own extensions using Python or C++. The Omniverse Kit comes with Python 3.6 and exposes many Python APIs. Therefore, we use this language to develop our Omniverse Applications.

Capturing a scene

Once you have the perfect image, use the capture scene button to save the image and metadata, which gives you everything you need for your project.

To create a synthetic dataset of more than a single image, Omniverse will automatically move the image, adjust lighting, etc., with a click of a button.

If you want to create a digital stream, Omniverse is compatible with RTSP streams. To output videos or get metadata from videos, simply get the data and plug it in. As well as Python, many other libraries are supported, such as Open CV.

Testing

The team at SmartCow currently uses Python tools for testing, which is great as this is familiar to most developers. However, Omniverse also has built-in testing tools which are worth exploring.

Collaboration

Because Omniverse is modular and easily extensible, team members with different skill sets can work seamlessly together on a project.

When multiple team members have worked on the code for a project, the approach at SmartCow is to consolidate the Python scripts team members have written and put them together in the final extension.

How easy is it to get started?

NVIDIA Omniverse allows creators, developers, and organizations to do many things. However, it is also complex, and it can take some time to learn the UI and API stacks to interact with the Omniverse.

For developers with a background in Python, the platform is easy to learn; however, the challenge is in finding the most efficient way to use it, which will require some research and learning. NVIDIA provides super-rich documentation where you can look up and copy examples of how to code various elements (e.g. a slider).

For 3D artists, the challenge is in learning how to integrate their 3D tools with Omniverse. However, if you’re not a 3D artist, there is no need to worry as Omniverse includes a super-rich catalog you can go through.

In summary, NVIDIA Omniverse is a rich platform which helps teams to collaborate and offers endless opportunities for organizations looking to work in the Metaverse space.

Watch the full webinar here.