NVIDIA Research Unveils Breakthroughs in Physical AI at SIGGRAPH

NVIDIA Research is at the forefront of advancing Physical AI, the technology powering next-generation robotics, self-driving vehicles, and intelligent environments. At SIGGRAPH, the leading computer graphics conference in Vancouver, NVIDIA researchers are showcasing how breakthroughs in graphics, simulation, and AI are converging to create realistic virtual worlds essential for training physical AI systems. Sanja Fidler, vice president of AI research at NVIDIA, emphasized the deep synergy between AI and graphics: “AI is advancing our simulation capabilities, and our simulation capabilities are advancing AI systems.” This integration, she noted, is a rare and powerful combination that few organizations can match. The company is unveiling several new tools and platforms at SIGGRAPH, including the NVIDIA Omniverse NuRec 3D Gaussian splatting libraries, which enable large-scale, high-fidelity world reconstruction from images and video. Updates to the NVIDIA Metropolis platform enhance vision AI for real-world applications, while new models like NVIDIA Cosmos Reason — a vision-language reasoning model — allow AI agents to apply human-like understanding, common sense, and physics-based reasoning to real-world tasks. Physical AI requires highly accurate 3D simulations to train robots safely and effectively. Without realistic virtual environments, skills learned in simulation would not transfer well to the real world. For example, a robot picking peaches must apply just the right pressure, or a factory robot must assemble tiny components with extreme precision. These tasks demand virtual worlds that mirror real-world physics and dynamics. Ming-Yu Liu, vice president of research at NVIDIA, explained that creating such worlds requires a blend of real-time rendering, computer vision, physics simulation, generative AI, and AI reasoning — all areas where NVIDIA has built deep expertise over nearly two decades. The company’s legacy in ray tracing and real-time graphics, dating back to 2006, underpins the visual fidelity needed for these simulations. AI is now central to both forward rendering — turning 3D scenes into 2D images — and inverse rendering — reconstructing 3D worlds from 2D input like photos or videos. Aaron Lefohn, head of the Real-Time Graphics Research group, highlighted that today’s AI can take everyday media and rapidly build detailed 3D environments, making high-fidelity simulation more accessible. NVIDIA’s Spatial Intelligence Lab introduced ViPE (Video Pose Engine), a new 3D geometric annotation system that analyzes videos from consumer cameras, dashcams, or film to estimate camera motion and generate accurate depth maps. This enables the creation of detailed 3D scenes from unstructured footage. Meanwhile, Liu’s Deep Imagination Research group is developing models that allow physical AI to predict future states — such as the outcome of a car running a red light or a glass tipping over — by combining visual understanding with physics and common sense. These innovations support NVIDIA Cosmos, a platform designed to accelerate physical AI with world foundation models, post-training tools, and efficient data pipelines. At SIGGRAPH, NVIDIA researchers are presenting over a dozen papers on neural rendering, real-time path tracing, synthetic data generation, and reinforcement learning. One key paper addresses a major challenge: ensuring that 3D models generated from 2D footage are not only visually accurate but also physically stable. Without this, simulated objects like chairs may collapse in a physics engine, undermining training reliability. This work bridges forward and inverse rendering to extract accurate physical parameters, ensuring synthetic data is not just realistic but also functionally valid. The special address by Fidler, Lefohn, and Liu at SIGGRAPH offers a deep dive into how graphics and simulation are driving the future of industrial digitalization. For those interested, the event runs through Thursday, August 14, and provides a window into the technologies shaping the next era of AI-powered physical systems.