SPEED-Bench unveiled as unified benchmark for speculative decoding
Researchers have introduced SPEED-Bench, a unified benchmark designed to evaluate speculative decoding, a critical technique for accelerating large language model inference. Speculative decoding utilizes a lightweight draft model to predict future tokens, which are then verified in parallel by a larger target model to improve throughput without altering output distribution. Despite the technique's rapid adoption, existing evaluation methods remain fragmented, often relying on small datasets, limited semantic diversity, and system configurations that do not reflect real-world production environments. SPEED-Bench addresses these gaps by combining purpose-built dataset splits with a unified measurement framework integrated into production-grade inference engines like TensorRT-LLM, vLLM, and SGLang. The benchmark evaluates speculative decoding from two distinct perspectives. The first is the Qualitative split, which prioritizes semantic diversity to measure draft accuracy across 11 distinct categories, including coding, mathematics, humanities, and roleplay. By using a custom selection algorithm to minimize semantic redundancy among 880 curated prompts, the dataset effectively exposes domain-dependent behaviors where low-entropy tasks like coding yield higher acceptance rates than high-entropy tasks like creative writing. The second component is the Throughput split, constructed to assess system-level performance under realistic serving conditions. This dataset aggregates prompts into fixed input sequence length buckets ranging from 1,000 to 32,000 tokens, reflecting the growing demand for long-context applications. It supports high concurrency and large batch sizes up to 512, allowing researchers to analyze the trade-offs between memory-bound and compute-bound regimes. A key finding from this split is that using random tokens to simulate load, a common practice in other benchmarks, significantly overestimates throughput by approximately 23% and fails to trigger realistic expert routing in mixture-of-experts models. The unified measurement framework standardizes evaluation by handling tokenization and prompt formatting externally, ensuring all systems process identical inputs. This isolation eliminates preprocessing artifacts that often skew cross-engine comparisons. Initial results from the benchmark reveal that speculative decoding performance varies significantly based on the model architecture and domain. For instance, native Multi-Token Prediction heads demonstrated superior acceptance lengths compared to post-trained alternatives like EAGLE3. Furthermore, the benchmark exposed hidden vulnerabilities in aggressive optimizations; vocabulary pruning in EAGLE3 showed negligible impact on coding and math tasks but caused substantial performance degradation in multilingual and retrieval-augmented generation categories. By providing a rigorous, diverse, and production-aware evaluation standard, SPEED-Bench aims to resolve the inconsistency in current speculative decoding research. The dataset and measurement tools are now openly available to the community, enabling developers and researchers to compare algorithms more accurately and optimize systems for real-world deployment. This initiative is expected to drive more reliable advancements in large language model serving efficiency.
