Summary: NVIDIA has significantly upgraded its physical AI foundation models with the release of Cosmos Predict-2. These models are crucial for building smarter robots and autonomous vehicles (AVs) by generating realistic, physics-aware future world states. Cosmos Predict-2 introduces several enhancements over its predecessor, including faster performance, improved visual quality, and greater customization capabilities, making it a versatile tool across various use cases and hardware platforms. Key Features and Improvements: Speed and Scalability: Cosmos Predict-2 offers two variants tailored to different levels of task complexity. The 2B model is ideal for quick prototyping and low-latency applications, capable of generating image previews in under 5 seconds on NVIDIA GPUs such as the GB200 NVL72, DGX B200, and RTX PRO 6000. The 14B model, while requiring more processing power, significantly enhances the quality and temporal coherence of the generated videos, ensuring high fidelity even in complex scenarios. Customization: One of the standout features of Cosmos Predict-2 is its ability to be customized for domain-specific use cases. Developers can generate a preview using the text-to-image model, which then conditions the video2world model to produce realistic, physically accurate world states. This process accelerates iterative prompting and scenario design, allowing for a wide range of applications, from robotics to industrial automation. Use Cases and Applications: Robotics: In robotics, Cosmos Predict-2 can be post-trained for instruction control and object manipulation tasks. For example, it can help a robot arm pick apples, taking into account variations in stem strength and lighting conditions. The post-training workflow involves collecting and curating 100 hours of teleoperation video, creating text plus visual pairing datasets, training the model, and validating the generated synthetic data using Cosmos Reason, an interpretation and reasoning model. Autonomous Vehicles (AVs): For AVs, the model can simulate rare and edge-case scenarios, such as rainy highway driving with synchronized lidar and camera data. Post-training involves generating diverse driving videos conditioned on HDMaps, LiDAR depth, and text prompts, ensuring realistic scenes under various conditions. This helps in creating robust training datasets that cover a wide range of driving environments. Industrial Automation: In industrial settings, Cosmos Predict-2 can be adapted for action-conditioned workflows, such as predictive maintenance for conveyor belt robots. The process is similar, involving data collection, post-training, and validation to ensure the synthetic data accurately represents real-world conditions. Vision: For vision applications, the model can generate 3D-consistent videos from single images by conditioning on camera poses. This is particularly useful for creating realistic scenarios in virtual environments, enhancing the quality and consistency of visual data. Post-Training Workflow: Prepare the Data: Collect approximately 100 hours of teleoperation video. Segment the video into relevant clips using data curation tools. Ensure the data reflects the specific setup, including robot models, lighting, and object types. Pair the video data with appropriate text captions using visual language models like Cosmos Reason. Post-Train the Model: Use the curated video-text pairs to fine-tune Cosmos Predict-2 for specific tasks and environments. Utilize post-training scripts available in the NVIDIA-Cosmos GitHub repository. Generate Synthetic Scenarios: Prompt the model with detailed text descriptions, such as "Pick up the bruised apple under low light." Optionally, use an initial image to create domain-specific "dream" videos. Validate for Physical Accuracy: Employ Cosmos Reason to evaluate the generated synthetic data for physical accuracy and common sense. For example, it can critique and optimize the "dreams" by ensuring actions like stopping at a stop sign are realistic and contextually appropriate. Integration with Other NVIDIA Models: Cosmos Predict-2 works seamlessly with other world foundation models within the NVIDIA Cosmos family. Cosmos Reason provides physical AI reasoning, interpreting visual input and performing chain-of-thought reasoning to generate accurate text decisions. Cosmos Transfer allows for dataset augmentation by adding variety to the synthetic data, such as different environments or lighting conditions, based on structured inputs or simulations created in NVIDIA Omniverse. Real-World Applications: NVIDIA Research has already demonstrated the potential of physical AI foundation models with Cosmos Predict-1. The DiffusionRenderer method, integrated into Cosmos, combines high-quality synthetic data and real-world video to improve lighting realism, geometry, and material accuracy in long video sequences. Difix3D+ enhances 3D reconstruction and novel view synthesis, improving temporal consistency and reducing flicker. The Cosmos-Drive-Dreams pipeline, based on Cosmos Transfer and Predict-1, generates diverse driving scenarios, extending from single-view to multi-view consistent videos. Getting Started: Developers can experiment with Cosmos Predict-2 by visiting the NVIDIA-Cosmos GitHub repository, which includes inference and post-training scripts for running open model checkpoints from Hugging Face. Subscribing to NVIDIA news and connecting with the Omniverse Developer Community can provide additional resources and updates on the latest physical AI advancements. NVIDIA also offers developer starter kits to quickly develop and enhance applications and services. Industry Evaluation and Company Profile: Industry insiders hail Cosmos Predict-2 as a game-changer in the realm of synthetic data generation for physical AI systems. The model's ability to generate high-fidelity, physics-aware data at a faster rate and with greater customization options is seen as a significant step forward in accelerating the development of smarter robots and AVs. NVIDIA, a leader in GPU technology and AI research, continues to push the boundaries of what is possible with physical AI models, underscoring its commitment to innovation and the advancement of intelligent machines. The integration of Cosmos Predict-2 with other NVIDIA models and its applicability across various domains highlight its robustness and versatility, marking it as a valuable tool for researchers and developers in the tech industry.