NVIDIA Recovery Automates Workload Recovery for Distributed AI Training

NVIDIA has introduced NVIDIA Recovery, an open-source workload-level service within NVIDIA DSX OS designed to keep large distributed AI training jobs running by automating failure recovery across clusters. As training scales, interruptions from NCCL timeouts, unresponsive GPU nodes, hangs, crashes and stragglers can halt progress and waste GPU hours. Manual recovery requires engineers to analyze logs, identify faulty resources and restart jobs from checkpoints, a process that becomes unsustainable when jobs run around the clock across large clusters. NVIDIA Recovery addresses this by running a cluster-wide control loop that monitors application logs, GPU metrics and scheduler signals, attributes failures, isolates implicated nodes and coordinates restarts from checkpoints on healthy capacity. The service integrates with the broader AI factory resilience stack. In Kubernetes environments, NVSentinel can monitor GPU health and drain faulty nodes, while Slurm sites may use custom health checks. Inside individual jobs, NVRx handles hang detection and in-job recovery. NVIDIA Recovery sits above these layers, focusing on cross-workload lifecycle management, placement and node exclusion. It is scheduler-agnostic through an Advisor API and currently supports Slurm, including Slurm on Kubernetes via slurm-operator, with native Kubernetes recovery planned for JobSet and Kubeflow PyTorchJob. For Slurm, a reference advisor controller can run in-cluster or over SSH, allowing deployment without changing scheduler internals or application code. NVIDIA highlighted production data from one training cluster between July 9 and October 9, where the system monitored nearly 200 jobs per day and detected about 8,550 anomalies. Mean detection latency was 69 seconds and recovery-action latency was 94 seconds. The company said the loop is especially valuable for very large jobs, where a single interruption affects many GPUs, and for clusters with many concurrent jobs, where operator overhead accumulates. Early adopters include IBM Research, which evaluated the system in monitor mode before enabling autonomous recovery, and Verda, the first NVIDIA Cloud Partner deployment, which runs the service with VictoriaLogs and contributed to the open-source project. NVIDIA Recovery starts in monitor mode by default, publishing recommended actions without changing scheduler state. Operators can compare those recommendations with manual practices before enabling action mode. All decisions are recorded in a persistent audit trail, while the Recovery Console shows workload state, anomaly flags and recovery timelines. The service is available in the dsx-ai-factory/NVIDIA-Recovery repository under the Apache 2.0 license. Future work focuses on Slurm job arrays, native Kubernetes workload recovery, sharper failure classification and a stable release. The goal is to reduce manual intervention and preserve training goodput as AI factories grow more complex.