HyperAIHyperAI

Command Palette

Search for a command to run...

Recreate Real-World Scenes in NVIDIA Isaac Sim Using Just a Smartphone and AI-Powered Tools

Reconstructing a real-world scene in NVIDIA Isaac Sim using only a smartphone is now possible thanks to NVIDIA Omniverse NuRec and the 3DGUT reconstruction pipeline. This workflow enables users to go from capturing photos on an iPhone to deploying a fully simulated robot environment—all without specialized hardware. Start by capturing the physical scene using your smartphone. Walk slowly around the area, taking photos with consistent lighting and sharp focus. Avoid fast movements or blurry images. Aim for about 60% overlap between consecutive shots and capture views from different heights and angles. If your iPhone saves images in HEIC format, switch to JPEG in Settings under Camera > Formats or convert the files before processing. Next, use COLMAP, an open-source Structure-from-Motion tool, to generate a sparse 3D reconstruction. Import your images into COLMAP and run the automatic reconstruction. Choose either the pinhole or simple pinhole camera model for compatibility with 3DGUT. COLMAP will produce a sparse point cloud and estimate camera poses. The output includes a database.db file, an images folder, and a sparse reconstruction directory—essential inputs for the next step. Now comes the core of the process: dense 3D reconstruction using 3DGUT. Run the 3DGUT training command with your COLMAP data and images. The algorithm will optimize a photorealistic 3D representation using Gaussian splatting. Depending on scene complexity and GPU power, this may take minutes to hours. Once complete, enable the export_usdz.enabled and export_usdz.apply_normalizing_transform flags to generate a USDZ file. This file packages the entire 3D scene with all visual and spatial data, ready for use in Isaac Sim. Open NVIDIA Isaac Sim (version 5.0 or later) and start with a blank stage. Import the USDZ file by selecting File > Import or by dragging it into the content browser. The scene will appear as a dense collection of colored points—Gaussian splats—recreating the real environment with high fidelity. Since the reconstructed scene contains only visual geometry, add a ground plane for physics. Use Create > Physics > Ground Plane to place a flat surface at the floor level. Scale it appropriately (e.g., 100x100 units) and align it with the reconstructed floor. Then, assign this plane as a proxy mesh so shadows can be cast correctly. In the Raw USD Properties panel under NuRec/Volume, add the ground plane to the Proxy field and ensure the Geometry > Matte Object property is enabled. Finally, insert a robot. Go to Create > Robots and select a SimReady model—such as the Franka Emika Panda, LeatherBack, Carter, TurtleBot, or a humanoid. Place the robot in the scene using the Move and Rotate tools. Once positioned, press Play to run the simulation. The robot now exists in a photorealistic, real-world environment, ready for tasks like motion planning, reinforcement learning, or testing perception systems. This end-to-end pipeline demonstrates how accessible high-fidelity robotics simulation has become. With just a smartphone and open-source tools, developers can build complex, real-world-like environments in Isaac Sim—accelerating innovation in AI and robotics.

Related Links