Run Edge AI Models in Few Commands With DGX Spark and TensorRT Edge-LLM

NVIDIA has positioned its newly available 64GB DGX Spark system as a development platform for physical AI applications that need to run large language models and agent workloads at the edge. As developers build systems for driver monitoring, robotics perception, task planning, and machine coordination on production lines, they increasingly require models that can perceive, reason, and act under tight memory, power, and latency constraints. DGX Spark 64GB is intended to help embedded developers prototype, fine-tune, test, and deploy such models, while NVIDIA TensorRT Edge-LLM provides a lightweight C++ inference runtime for these workloads. A central part of the workflow is the trtedge command-line interface, which reduces model setup to a few commands. The tool can download a selected model, build TensorRT inference engines optimized for Spark, and cache the results for later use. The tutorial demonstrates this process with NVIDIA Nemotron-3.5 Lightning, a text-only model. Developers can run it in a terminal chat mode, serve it through a local OpenAI-compatible API, and use the API to prototype applications before targeting embedded platforms such as NVIDIA Jetson Thor, NVIDIA DRIVE AGX Thor, and NVIDIA IGX Thor. The environment requires DGX Spark with a supported NVIDIA software stack, NVIDIA CUDA 13, TensorRT 10.16.1.11 or a newer 10.x release, and Python 3.10 to 3.12. The example uses TensorRT Edge-LLM 0.11.1. After creating and activating a Python virtual environment, users install the runtime, verify TensorRT compatibility, and launch the model. The first run downloads approximately 21GB of model data and builds the inference engine, while later runs can reuse cached engines. For application testing, the model can be served through a local OpenAI-compatible API, with status checks and chat completion requests supported through JSON streaming responses. NVIDIA also highlights optional performance gains from DSpark speculative decoding, which uses a draft model to propose multiple tokens for verification by the main model in a single pass. In the provided results on DGX Spark 64GB, Nemotron-3.5 Lightning generation speed increases from about 73 tokens per second with speculative decoding disabled to roughly 105 to 127 tokens per second with DSpark enabled, depending on the workload. Peak total system memory use during inference with DSpark was reported at 27.9GB, including the operating system. NVIDIA said developers can move from Spark-based prototyping to a broader set of edge scenarios by exploring supported vision-language, speech recognition, and text-to-speech models. The company also pointed users to TensorRT Edge-LLM documentation for platform-specific requirements and to Nemotron developer resources for open models, datasets, and application blueprints.