NVIDIA Launches Cosmos3-H-Dreams Interactive Generative Surgical Simulation

NVIDIA has introduced Cosmos3-H-Dreams, a real-time generative surgical simulator designed to turn clinical procedure recordings into interactive visual environments for surgical robotics research. The system advances NVIDIA’s earlier Cosmos-H-Dreams work, which used tabletop robot suturing tasks to test world foundation models as alternatives to numerical physics simulation. Cosmos3-H-Dreams moves from controlled benchtop demonstrations toward complex in-body surgical scenes, using recordings from clinical cholecystectomy procedures performed on the CMR Surgical Versius platform. According to NVIDIA, it is the first real-time generative surgical simulator trained on clinical cholecystectomy data to support live kinematic control. The simulator is powered by Cosmos 3 and runs interactively on a single NVIDIA RTX PRO 6000 GPU, with acceleration provided by FlashDreams, NVIDIA’s streaming-inference library. Instead of explicitly modeling geometry, tissue deformation and physical interactions, the system learns visual dynamics from synchronized surgical video and robot kinematics. When given an initial frame and instrument actions, it generates new video blocks in response to controls such as translation, rotation and gripper commands. NVIDIA says this enables developers to explore surgical interaction through generated scenes with anatomy, tissue motion, occlusion and changing lighting or smoke conditions. Cosmos3-H-Dreams is built on Cosmos3-H-Embodied-Nano, a 16-billion-parameter model pretrained on Open-H-Embodiment data, and further specialized on curated cholecystectomy video and kinematics. The training setup focuses on two engaged surgical instruments, a left Cadiere grasper and a right monopolar hook, with camera commands limited to emphasize instrument-driven interaction. The model uses 28 action dimensions, including arm poses, gripper state, energy controls and thumbstick inputs. Training uses 832 by 480 video frames with actions aligned at 10 Hz, while runtime generation is optimized for low-latency interactive use. To support long interactive rollouts, NVIDIA uses a teacher-to-student training pipeline. A bidirectional teacher is first adapted to longer horizons and then distilled into causal student models capable of generating short video blocks as new actions arrive. Two released checkpoints use either four or two denoising steps per block, with the two-step model separately trained for faster inference and reduced visual quality. FlashDreams manages streaming generation, while an adapted FadeMem module consolidates attention history during long sessions. NVIDIA says the simulator can generate coherent continuations from unseen movements and allows researchers to inspect whether instrument controls, tissue response and anatomical consistency remain stable over time. The system includes checkpoints and guidance for adapting the method to other datasets, embodiments and controllers that use compatible action conventions. The company emphasizes that Cosmos3-H-Dreams is a research tool, not a validated clinical system for patient care, and is intended to support exploration of generative simulation for surgical robotics and embodied AI development.