HyperAIHyperAI

Command Palette

Search for a command to run...

NVIDIA Releases Cosmos 3 Edge for Real-Time Robot Reasoning and Actions.

NVIDIA has released Cosmos 3 Edge, a 4-billion-parameter open world model designed to bring data center-level performance to memory-constrained edge devices. Available via the Hugging Face Cosmos 3 repository, this open world foundation model enables robots and vision AI agents to perceive environments, reason in real time, and generate actions directly on-site. The model targets physical AI applications in factories, warehouses, and hospitals, delivering best-in-class throughput and accuracy for vision analytics and robot policy learning. Cosmos 3 Edge operates on NVIDIA edge hardware, including RTX PRO and GeForce RTX GPUs, DGX systems, and the newly announced Jetson T2000 and T3000 modules. It processes observations at robot-control resolution of 640x360 pixels, generating 32 actions per inference on the Jetson Thor while maintaining real-time control at 15 Hz. Among models of comparable size, Cosmos 3 Edge ranks number one on the VANTAGE-Bench for vision analytics and achieves state-of-the-art results in robot policy learning. The architecture utilizes two Transformer towers sharing a common multimodal representation. An autoregressive tower provides reasoning tokens, while a diffusion tower generates video and action tokens. Both towers share attention layers that align language, video, audio, and action data, allowing the system to simulate possible futures and link predictions to physical outcomes. Actions are encoded as compact geometric vectors that unify diverse embodiments, such as vehicle poses, camera motion, and gripper states, into a unified control framework. This structure supports bidirectional inference, enabling the model to predict visual consequences from actions or infer optimal actions from observed changes. NVIDIA is also releasing post-trained checkpoints to accelerate deployment. The Cosmos 3 Edge Policy DROID offers a manipulation policy optimized for pick-and-place tasks using the DROID dataset. Additionally, the Cosmos 3 Super 4-Step Distillation checkpoint applies distribution-matching distillation to the 64B model, reducing diffusion denoising steps from 35 to 50 down to just four, yielding up to 25 times faster inference for text-to-image and image-to-video tasks. Developers can use provided scripts to fine-tune these models on H100 or DGX Station clusters before deploying optimized versions to edge platforms. Future roadmap items include enhanced interactive world generation, driving scenario simulation, and deeper integration with optimization frameworks like vLLM.

Related Links