News summary:
- The NVIDIA Agent Toolkit now includes NVIDIA Omniverse libraries and provides AI agents with tools and capabilities that enable software developers to integrate physical AI capabilities into their existing applications.
- New Omniverse libraries for NVIDIA RTX sensor simulation, GPU accelerated physics simulation, and simulation-ready asset validation are openly available on GitHub.
- SideFX and PTC integrate Omniverse libraries into 3D physical AI applications with support for cloud and on-premises AI systems, from NVIDIA RTX Spark to NVIDIA DGX Station.
- New NVIDIA blueprint to integrate Omniverse libraries into Blender.
NVIDIA today announced that the NVIDIA Agent Toolkit now includes NVIDIA Omniverse™ Libraries – a collection of software components that provide AI agents with tools and capabilities to add physical AI capabilities to existing applications and prepare 3D content for simulation.
Robots, factories and autonomous systems must be designed, tested and trained in simulations before being deployed in the real world. Preparing 3D content for simulation requires more than realistic images – assets need the right structure, materials, scaling, annotations, sensors and physical properties. With the NVIDIA Omniverse libraries in the NVIDIA Agent Toolkit, AI agents have the tools and capabilities to create workflows, inspect scenes, flag issues, and prepare assets. This allows developers to move more quickly from 3D content to simulation-ready environments.
“The era of physical AI will be built in simulation first,” said Jensen Huang, founder and CEO of NVIDIA. “NVIDIA Agent Toolkit with Omniverse Libraries brings AI agents into the 3D tools developers already use, helping build simulation-ready worlds where robots, factories and autonomous systems are trained and tested long before they reach the real world.”
Software vendors such as SideFX and PTC are integrating Omniverse libraries for agent-enabled sensor simulation, physics and asset validation, helping to integrate agentic AI into the applications and workflows that developers and technical artists already use to prepare 3D content.
Omniverse Libraries bring physical AI capabilities to the NVIDIA Agent Toolkit
The NVIDIA Agent Toolkit helps software manufacturers develop AI agents that connect tools, capabilities and data sources. Omniverse libraries extend these agents into 3D and physical AI workflows with callable tools for sensor simulation, GPU-accelerated physics, and simulation-ready asset validation in existing applications.
The new Omniverse libraries – including ovrtx, ovphysx, and CAD-to-SimReady capabilities – are openly available on GitHub and provide AI agents with tools to create workflows for inspecting scenes, testing changes, and preparing 3D assets for simulation. A new draft for integrating Omniverse libraries into Blender is now also available on GitHub.
The main functions of the libraries include:
- NVIDIA RTX sensor simulation: ovrtx helps applications generate camera, lidar, radar, and other sensor outputs from 3D scenes, allowing developers and AI agents to test how physical AI systems can perceive virtual environments.
- Physical behavior: ovphysx uses GPU-accelerated physics to bring realistic behavior to 3D scenes using properties such as collisions, mass, friction and motion, allowing teams to first test how objects and systems interact in the simulation.
- Simulation-capable 3D objects: CAD to SimReady capabilities help convert computer aided design (CAD) data into OpenUSD-based SimReady assets, giving 3D content the properties required for physical AI simulation and virtual testing.
Software manufacturers build with Omniverse libraries
Software manufacturers such as SideFX and PTC, as well as startups ForgeCAD, Lightwheel, Moonlake AI and Palatial, are among the first to adopt and create Omniverse libraries, which are now part of the NVIDIA Agent Toolkit.
SideFX uses OpenUSD workflows as well as ovrtx and ovphysx libraries to explore how Agents can help integrate Omniverse libraries into its Houdini 3D procedural content creation workflows, giving technical artists a way to generate physics, test physics, and prepare content for simulation.
“Procedural 3D creation is critical to building the complex, controllable worlds required for simulation, robotics and industrial AI,” said Kim Davidson, President and CEO of SideFX. “With NVIDIA Omniverse libraries and OpenUSD, SideFX explores how agent-ready tools can support Houdini workflows, helping technical artists review, test and prepare procedural content for simulation while maintaining control of the creative process.”
The PTC Onshape CAD and product data management (PDM) platform leverages OpenUSD and ovrtx to connect cloud-native design workflows with physical simulation, helping product design content stay connected to CAD, PDM, collaboration and simulation workflows.
“Development teams are looking for more connected ways to design, collaborate and simulate throughout the development process,” said Neil Barua, president and CEO of PTC. “PTC’s collaboration with NVIDIA supports this broader vision, while NVIDIA Omniverse libraries help enable simulation-ready workflows that bring validation and testing closer to where products are developed.”
SIGGRAPH will showcase “SimReady” Blender, a sample workflow built in Blender with NVIDIA Omniverse libraries and NVIDIA NemoClaw™ that shows how software manufacturers can integrate agent-ready simulation capabilities – including NVIDIA RTX sensor simulation, physics and validation – into existing 3D applications while maintaining control. This is now openly available as a blueprint for integrating Omniverse libraries into Blender.
The demo also previews how these workflows, built with Omniverse libraries as part of the NVIDIA Agent Toolkit, can be run locally, from compact RTX-based systems with NVIDIA RTX Spark™ to NVIDIA GB300-based systems with NVIDIA DGX Station™. RTX Spark systems will be available this fall from ASUS, Dell Technologies, HP, Lenovo, Microsoft Surface and MSI, with models from Acer and GIGABYTE to follow. DGX Station systems can be ordered from ASUS, Dell, GIGABYTE, HP, MSI, Supermicro and Exxact.
Startups, including those participating in the NVIDIA Inception program, are also leveraging Omniverse libraries and capabilities to add agent-based asset and scene preparation workflows. Palatial leverages Omniverse CAD-to-SimReady capabilities to automate the creation and validation of SimReady assets at scale from CAD inputs. Lightwheel uses Omniverse Content Agents based on OpenUSD in its SimReadyGen technology to generate physically accurate SimReady assets from text prompts.
ForgeCAD and Moonlake AI explore agent-driven 3D content workflows that leverage Omniverse capabilities to generate, extend, and prepare assets for physical AI simulation.
Watch them NVIDIA keynote at SIGGRAPH. Find out more about NVIDIA Omniverse Libraries and explore available examples and documentation.
https://nvidianews.nvidia.com/news/nvidia-agent-toolkit-expands-with-new-omniverse-libraries-putting-ai-agents-to-work-building-simulation-ready-worlds
