NVIDIA AIPerf Scales LLM Inference Benchmarking With Multiprocess Architecture.

NVIDIA has introduced AIPerf, a ground-up rewrite designed to address the persistent scalability and accuracy challenges in large language model inference benchmarking. Serving as the official successor to GenAI-Perf, the tool was engineered to eliminate client-side bottlenecks that frequently distort production-level performance metrics. By abandoning the single-process architecture that plagues legacy benchmarking utilities, AIPerf employs a multiprocess framework where dedicated worker processes generate load, while independent record-processor services aggregate results. Coordination across these components is handled through ZeroMQ, effectively bypassing the Global Interpreter Limit in Python and enabling the client to fully saturate target servers without compromising measurement fidelity. The framework supports over fifteen distinct endpoint types, encompassing chat completions, response generation, model rankings, and image synthesis. It also accommodates diverse traffic replay formats from providers such as Mooncake, Baseten, and WEKA, allowing engineers to transition seamlessly between synthetic smoke tests and reconstructed production workloads. Engineers retain granular control over traffic shaping through configurable arrival patterns, including constant, Poisson, and gamma distributions. These parameters allow for tunable burstiness, gradual concurrency ramping, and synthetic input-output sequence length variations that mirror real-world inference demands. To ensure precise latency and throughput measurements, AIPerf mandates streaming responses, which captures time-to-first-token and inter-token latency at the individual decode level rather than relying on fully batched outputs. Benchmark results are delivered through a standardized metrics table and exported to CSV or JSON formats, covering core performance indicators alongside comprehensive percentile breakdowns. This statistical granularity reveals long-tail latency distributions that aggregate averages often obscure. When integrated with DCGM or pynvml, the tool automatically correlates inference performance with hardware telemetry, including GPU power consumption, utilization rates, and memory pressure. Initial deployment workflows have been validated against vLLM-served models, demonstrating rapid configuration cycles and straightforward metric interpretation. The architecture prioritizes reproducibility, utilizing deterministic seeding for traffic patterns and explicit token constraints to maintain baseline consistency across iterative testing. Development reflects a broader industry push toward standardized evaluation methodologies, built through cross-company collaboration involving engineers from AWS, Coreweave, Baseten, and Pinterest. By unifying scalable load generation, flexible workload simulation, and comprehensive telemetry into a single utility, AIPerf establishes a new baseline for LLM inference validation, reducing engineering overhead while increasing confidence in deployment readiness.