HyperAIHyperAI

Command Palette

Search for a command to run...

기준
LLM

AscendGYM: Ascend NPU의 실제 워크로드를 위한 비용 인지형 커널 최적화 벤치마크

초록

커널 최적화는 Ascend NPU에서 대규모 언어 모델(LLM)의 효율적인 학습과 추론을 위해 필수적이다. 기존 Ascend 커널 벤치마크는 주로 커널 생성에 초점을 맞추고 있으며, 실제 워크로드에서 도출된 입력 구성 하에서 에이전트가 기존 구현을 최적화하는 능력에 대한 평가는 상대적으로 부족하다. 이러한 요구를 해결하기 위해 본 논문에서는 세 가지 난이도에 걸쳐 48개의 커널로 구성된 초기 스위트를 갖춘 벤치마크인 AscendGYM을 소개한다. 각 커널에는 실제 학습 및 추론 작업에서 수집된 데이터 유형과 텐서 형태를 가진 테스트 케이스가 짝지어져 있다. 따라서 AscendGYM에서의 성능 향상은 해당 워크로드에 직접적으로 대응된다. 사용성 측면에서 AscendGYM은 모델 평가와 모델-하네스 평가라는 두 가지 평가 모드를 지원한다. 이를 통해 학습된 모델과 사용자 정의 에이전트의 커널 최적화 능력을 모두 측정할 수 있다. 또한 실제 파이프라인에서 Ascend 커널이 호출되는 방식을 충실히 재현하는 완전한 실행 환경을 제공한다. 모델 평가의 경우 탐색 오케스트레이션과 경험 축적 기능을 갖춘 내장 하네스도 포함한다. 마지막으로 토큰 비용 대비 성능 향상을 정량화하는 새로운 지표인 Speedup–Token Score(STS)를 제안한다. STS는 속도 향상과 비용을 함께 고려함으로써 수용 가능한 예산 내에서 모델 또는 하네스의 실질적인 커널 최적화 능력을 평가할 수 있게 한다. AscendGYM은 https://github.com/SLAI-AITP/AscendGYM에서 이용할 수 있다.

One-sentence Summary

Researchers from Shenzhen Loop Area Institute introduce AscendGYM, a cost-aware Ascend NPU kernel optimization benchmark with 48 kernels across three difficulty levels and real-workload test cases, supporting model and model-harness evaluation via a built-in harness and proposing the Speedup-Token Score to assess practical kernel optimization gains relative to token cost.

Key Contributions

  • The paper introduces AscendGYM, a kernel optimization benchmark for Ascend NPUs with 48 kernels across three difficulty levels, complete runnable initial source code, and input configurations drawn from real LLM training and inference workloads.
  • It provides AscendForge, an execution environment for candidate compilation, correctness checking, and performance measurement, supporting both model evaluation with a built-in harness and model-harness evaluation.
  • It proposes the Speedup-Token Score (STS), a metric that combines performance gains with cumulative token cost, and reports Batch 1 results on 17 tasks showing the benchmark remains challenging and STS reveals differences that speedup alone does not capture.

Introduction

Efficient large language model training and inference depend heavily on low-level kernel performance, and on Ascend NPUs optimizing existing kernels for real workloads is a critical engineering task. Prior benchmarks primarily evaluate whether models can generate correct kernels from task specifications, but they do not systematically assess iterative optimization of already correct implementations under workload-derived input configurations and explicit resource budgets. The authors introduce AscendGYM, a benchmark with 48 kernel optimization tasks drawn from real LLM training and inference workloads. It provides runnable baseline implementations, workload-matched input shapes and data types, an execution sandbox called AscendForge for compilation, correctness checking, and performance measurement, and a Speedup-Token Score that jointly captures achieved speedup and token consumption. Evaluation on Batch 1 shows most current agents achieve only modest gains, confirming that optimizing existing Ascend kernels remains challenging.

Dataset

The authors describe AscendGYM as a kernel optimization benchmark for Ascend NPUs, not as a model training dataset. It is designed to evaluate whether model-harness systems can improve existing kernel implementations under consistent rules.

Dataset composition and sources

  • The benchmark has four main components:
    • Kernel optimization task set and test cases
    • AscendForge evaluation environment and tools
    • Evaluation metrics
    • Shared evaluation workflow
  • Tasks are sourced from bottleneck kernels that have substantial performance impact in real workloads.
  • Tasks are organized into three difficulty batches:
    • Batch 1: 17 basic kernels, focused on memory access optimization
    • Batch 2: 17 tasks covering multistep computations such as normalization and quantization
    • Batch 3: 14 tasks covering complex attention mechanisms and fused computations, including sparse attention
  • The current paper evaluates only the 17 Batch 1 tasks. Batch 2 and Batch 3 results are left for future updates.

Test case design and metadata

  • Each test case preserves the connection between kernel inputs and deployment context.
  • The context includes:
    • Target model
    • Execution phase
    • Hardware and parallelism configurations
    • Tensor shapes
    • Data types
    • Computational parameters
  • For example, the AddRMSNorm task captures residual addition followed by RMS normalization in a forward pass of Qwen3-0.6B inference.
  • That test case uses:
    • Prefill phase
    • No quantization
    • Eager execution
    • Single NPU
    • Tensor parallelism 1
    • BF16 precision
    • Batch size 1
    • Input sequence length 32 tokens
  • Profiling metadata for AddRMSNorm reports 56 invocations per forward pass, accounting for 8.77 percent of total computation time and ranking among the top-5 kernels by cumulative execution time.

Processing and evaluation usage

  • The benchmark provides tasks and execution feedback to assess optimization outcomes.
  • Evaluation covers multiple deployment cases, best achievable performance for specific shapes and scenarios, and optimization cost.
  • These dimensions are used to characterize:
    • Cross-case applicability
    • Shape specialization
    • Optimization efficiency
  • No training split or data mixture ratio is reported, because AscendGYM is used for evaluation rather than model training.

Method

AscendGYM organizes kernel optimization tasks into three batches of increasing difficulty, progressing from largely independent kernels to multistage fusion and complex execution coordination. Batch 1 focuses on memory access optimization for basic kernels, Batch 2 covers multistep computations like normalization and quantization, and Batch 3 targets complex attention mechanisms and fused computations. Task selection prioritizes bottleneck kernels with substantial performance impact in real workloads. Each test case preserves the connection between kernel inputs and their deployment context, including the target model, execution phase, hardware and parallelism configurations, and resulting tensor shapes. This context links measured kernel speedups to corresponding operator invocations in real workloads, aligning evaluation with deployment requirements.

AscendGYM includes AscendForge, an Ascend-native execution sandbox for candidate compilation, correctness checking, and performance measurement. Building on SandboxFusion, AscendForge introduces adapters that manage dependencies among host-side registration, device-side kernels, the target system-on-chip, and custom kernel packages, enabling compilation and Python interface binding for NPU devices.

As shown in the figure below:

AscendForge comprises four layers for kernel building and execution, resource scheduling, development feedback, and deployment and operations. The Execution and Build Layer supports Ascend C and Python NPU execution backends, with adapters for CANN kernel repositories and projects like vLLM Ascend. It automates source submission, compilation and installation, Python interface binding, and test execution. The Resource Orchestration Layer manages evaluation requests through a scheduling proxy, task queues, and multiple execution nodes, allocating resources according to the required NPU count and device model. The Development Feedback Layer integrates static code analysis and performance profiling, returning build logs, correctness results, and performance measurements to help agents diagnose errors and guide the next optimization iteration. The Deployment and Operations Layer provides hardware health monitoring, configuration hot reloading, and automatic process recovery.

The authors leverage joint performance-budget metrics, standalone performance and budget metrics, and a reliability metric to evaluate optimization outcomes and the resources required to achieve them.

The Speedup-Token Score (STS) jointly measures the best speedup achieved within a token budget and the token consumption required to obtain improvements:

STS=100log⁡2Sn⏟final performance gain−50∑k=1nBkBmax⁡Δgk⏟penalty for delayed improvements\text{STS} = \underbrace{100 \log_2 S_n}_{\text{final performance gain}} - \underbrace{50 \sum_{k=1}^n \frac{B_k}{B_{\max}} \Delta g_k}_{\text{penalty for delayed improvements}}STS=final performance gain100log2​Sn​​​−penalty for delayed improvements50k=1∑n​Bmax​Bk​​Δgk​​​

Here, Bmax⁡>0B_{\max} > 0Bmax​>0 is the token budget, nnn is the number of candidate evaluations completed within that budget, and BkB_kBk​ denotes the cumulative token consumption when evaluation kkk completes. SkS_kSk​ denotes the best correctness-validated speedup available after evaluation kkk, relative to the reference implementation. The incremental log-speedup gain is Δgk=log⁡2Sk−log⁡2Sk−1\Delta g_k = \log_2 S_k - \log_2 S_{k-1}Δgk​=log2​Sk​−log2​Sk−1​.

The Cost-to-Gain (CTG) metric measures the additional token consumption required to increase speedup by a specified fraction relative to a reference checkpoint:

CTG⁡(δ;r)=min⁡r≤k≤nSk≥(1+δ)Sr(Bk−Br),δ∈{0.01,0.05,0.10}\operatorname{CTG}(\delta; r) = \min_{\substack{r \leq k \leq n \\ S_k \geq (1+\delta)S_r}} (B_k - B_r), \qquad \delta \in \{0.01, 0.05, 0.10\}CTG(δ;r)=r≤k≤nSk​≥(1+δ)Sr​​min​(Bk​−Br​),δ∈{0.01,0.05,0.10}

Lower CTG indicates greater token efficiency.

For standalone performance, AscendGYM reports GM Speedup and the fastp\text{fast}_pfastp​ metric. The fastp\text{fast}_pfastp​ metric measures the fraction of tasks for which a correct implementation achieves a speedup strictly greater than ppp:

fastp=1N∑i=1N1[Si,ni>p],p≥1\text{fast}_p = \frac{1}{N} \sum_{i=1}^N \mathbb{1}[S_{i, n_i} > p], \qquad p \geq 1fastp​=N1​i=1∑N​1[Si,ni​​>p],p≥1

GM Speedup summarizes performance across tasks using their geometric mean:

GM Speedup=(∏i=1NSi,ni)1/N\text{GM Speedup} = \left( \prod_{i=1}^N S_{i, n_i} \right)^{1/N}GM Speedup=(i=1∏N​Si,ni​​)1/N

Budget metrics measure optimization cost using cumulative and mean token consumption, LLM interaction rounds, and wall-clock time. Reliability metrics include the Candidate Pass Rate, which measures how reliably an LLM or agent harness produces correct candidates, and the Task Pass Rate, which measures the fraction of attempted tasks with at least one candidate that passes correctness checks.

AscendGYM supports two evaluation modes: model-harness evaluation, which evaluates a model-harness pair as a complete system, and model evaluation, which runs the model with a default harness called AuraKernel.

As shown in the figure below:

The workflow constructs kernel optimization tasks from real training and inference workloads. Profiling data from workloads running in Ascend frameworks are collected to identify the top-k bottleneck kernels. Each kernel optimization task includes the source code of the baseline, template metadata specifying input and output tensor shapes, test inputs, and reference outputs generated by the corresponding CPU implementation. To launch an evaluation, users configure the execution environment and optimization settings. AscendGYM first measures baseline performance, then enters an iterative loop of bottleneck analysis, code refinement, performance measurement, and experience summarization. Each optimized candidate is submitted to the AscendForge sandbox for compilation, Python interface binding, and correctness checks. Candidates that pass these checks are profiled, and the resulting measurements guide subsequent optimization. Search terminates when the resource budget is exhausted or the maximum iteration count is reached. The system then revalidates the best implementation and uses the performance measurements and optimization traces to compute the defined metrics.

Experiment

AscendGYM evaluates NPU kernel optimization through AscendForge sandbox execution and two modes: model-harness pairs using Claude Code, Codex CLI, and OpenCode, and model-only evaluation with the Default AuraKernel harness. Joint speedup-token and cost-to-gain metrics, along with performance, budget, and reliability measures, are used to compare approaches under 10M, 20M, and 50M token budgets. The experiments show that final speedup alone hides differences in when gains are achieved and at what cost, while pass rates and aggregate speedup can diverge. Current systems mostly achieve modest improvements, with around 70% of model-harness combinations below 1.2x geometric mean speedup, and the warm Default configuration performs best in model evaluation, indicating that both harness configuration and budget constraints materially affect optimization outcomes.


AI로 AI 구축

아이디어에서 출시까지 — 무료 AI 코코딩, 즉시 사용 가능한 환경, 최적의 GPU 가격으로 AI 개발을 가속화하세요.

AI 협업 코딩
바로 사용 가능한 GPU
최적의 가격

HyperAI Newsletters

최신 정보 구독하기
한국 시간 매주 월요일 오전 9시 에 이번 주의 최신 업데이트를 메일로 발송합니다
이메일 서비스 제공: MailChimp