NVIDIA Nemotron Voice Agent Powers Interactive Haunted House

An NVIDIA creator developed MonsterHouse, an interactive Halloween projection that gives a garage door a speaking, animated face, using NVIDIA Nemotron open models and related tools to deliver a self-contained voice agent on a single workstation. The installation lets trick-or-treaters converse with a grumpy house voiced as a pirate ship reincarnated as real estate. The system listens to visitors, answers with nautical complaints, and softens when they say please or thank you, distributing candy through an automated greeting and farewell flow. The build combines the NVIDIA Nemotron Voice Agent as the orchestration layer, Nemotron Speech for listening, Nemotron 3.5 Bolt E4B 4-bit as the local language model, Magpie TTS Zeroshot for voice generation and Audio2Face-3D for mouth animation. A wooden door rig was created in Autodesk Maya and animated in Unreal Engine with flame effects. The character has no eyes or blinking, reducing visual complexity while preserving expression through a split-door mouth. To give the character a consistent tone, the developer trained low-rank adapters on a dataset of more than 5,000 reviewed multi-turn conversations, using an agent and large model to generate and evaluate data. The workflow was intended to track visitor state, such as whether manners had been used, while leaving compute for speech and graphics. Testing selected Bolt E4B over larger Qwen alternatives because of its smaller footprint and suitability for local conversational characters. Performance was improved through text caching and saved audio for common greetings, jokes and replies, allowing the house to begin speaking before a full response is generated. Median end-of-speech-to-audio latency was 1.70 seconds with live generation and 0.72 seconds for cached responses, with 300 milliseconds of silence detection. Magpie Zeroshot used short reference samples for seven moods, while Audio2Face-3D drove the mouth from clean speech and separate effects added deep, echoing vocal treatment. The pipeline ran on two GPUs, with an RTX PRO 6000 Blackwell Max-Q handling language and voice models and an RTX PRO 4000 Blackwell running Unreal Engine and face animation. A control desk provides audition, model selection, dialogue overrides, emotion tuning, conversation monitoring, camera feeds and performance modes, including a microphone fallback that lets the operator speak through the house. Future work includes Nemotron 3 Nano Omni for visual costume recognition and lightweight YOLO and ByteTrack counters. The developer said models remain available, though MonsterHouse code is not packaged, and recommends a path from Nemotron Voice Agent to Bolt, Magpie and Audio2Face for custom character builds.