NVIDIA Releases Open Cosmos 3 Models for Interactive World Simulation

NVIDIA has released four open-weight checkpoints for its Cosmos 3 world-model development platform, extending the Cosmos 3 Nano family with models designed for interactive world simulation and multi-sensor driving-scene generation. The checkpoints, each with 16 billion trainable parameters, are post-trained examples available under the OpenMDW1.1 license, which permits commercial and non-commercial use. The release is intended to give robotics and autonomous-driving developers a shared base for generating, adapting, and evaluating simulated environments across changing scenes, viewpoints, and control inputs. The portfolio includes Cosmos3-Nano-Transfer-Auto, a bidirectional offline generator for synthetic driving data. It can produce synchronized video from up to 11 camera views, including fisheye cameras, and supports joint RGB and LiDAR generation. Developers can guide scene structure and traffic motion with per-camera World Scenario Map controls, use text prompts for appearance, and anchor initial frames with reference images. Aligned HD-map range-map controls can additionally condition LiDAR synthesis. NVIDIA positions Transfer-Auto as an adaptation starting point for custom sensor rigs and driving domains; after post-training on domain data, it can be used directly for data generation or distilled into a faster streaming model. That streaming variant, Cosmos3-Nano-Sim-Auto, is the causally distilled version of Transfer-Auto. It produces successive short chunks of synchronized camera video and LiDAR in a small number of denoising steps, using rolling generated history and updated driving controls. This enables longer interactive rollouts and closed-loop testing when paired with a simulation runtime and World Scenario Map renderer that updates controls from driving inputs. For robotics, NVIDIA introduced Cosmos3-Nano-Sim-Transfer and Cosmos3-Nano-Sim-Bimanual. Sim-Transfer converts spatial controls from a physics simulator into RGB observations, allowing a policy to act while the simulator preserves ground-truth motion, contact, and physical state. Sim-Bimanual generates the next visual observation directly from a bimanual robot's actions, supporting interactive two-arm simulation for YAM and AgiBot setups under policy or joystick control. It handles mixed viewpoint-change and manipulation sequences using a defined action schema and normalized control values. The models rely on Cosmos 3’s Mixture-of-Transformers architecture, linking text-processing and visual-generation networks through attention. During generation, the system uses available controls, prompts, and generated history to predict compressed video latents, where each latent time step represents four RGB frames. Temporal-causal attention and key-value caching support streaming inference. Training includes self-forcing, distillation, and caching to reduce compounding errors and inference cost. NVIDIA also released post-training recipes covering the full training path, along with checkpoint downloads, technical reports, demos, and sample skills for inference and adaptation. The release underscores NVIDIA’s push to provide an open foundation for simulated environment research, though the company notes that measuring fidelity for policy learning, evaluation, and real-world adaptation remains an open challenge. Users need Linux, an NVIDIA GPU, Docker, and the NVIDIA Container Toolkit to run the published vLLM-Omni setup, which was tested on a single GB200 but is not presented as a minimum hardware requirement.