One-sentence Summary

SLAI T-Rex, developed by the AI Training Platform Team at Shenzhen Loop Area Institute Models Codes, is an Ascend SuperPOD-optimized framework that post-trains trillion-parameter DeepSeek-V4 family MoE models to $34.22\%$34.22% MFU (a $2.93\times$2.93× improvement) and couples solver-grounded CPT-SFT to specialize for Operations Research, achieving $71.81\%$71.81% zero-shot Pass@1, outperforming GPT-5.4-Mini by $3.98$3.98 points and the base DeepSeek-V4-Flash model by $11.27$11.27 points.

Key Contributions

• A hierarchical optimization framework for full-parameter post-training of trillion-scale MoE models on the Ascend NPU SuperPOD achieves 34.22% model FLOPs utilization and a 2.93× improvement over the open-source baseline while maintaining training stability.
• SLAI T-Rex couples this Ascend SuperPOD optimization with a solver-grounded continued pre-training and supervised fine-tuning pipeline for operations research, built on a curated 10K-sample SFT dataset covering four task categories and three problem representations.
• The specialized DeepSeek-V4-Flash model reaches a 71.81% average zero-shot Pass@1 score, outperforming GPT-5.4-Mini by 3.98 percentage points and the base model by 11.27 percentage points, and continued pre-training establishes transferable OR-domain priors that improve solver-facing feasibility and structural equivalence after fine-tuning.

Introduction

The authors address the post-training of large language models (LLMs) for agentic tasks, a process that typically relies on CUDA- or TPU-based infrastructure and focuses on domains like mathematics and coding. Prior work largely neglects training trillion-parameter mixture-of-experts (MoE) models on single-instruction, multiple-data (SIMD) hardware such as Ascend NPUs, and it barely touches domain-specific adaptation for Operations Research (OR), a field central to scheduling, supply chains, and resource allocation. The remaining OR approaches suffer from brittle formulation mapping, small benchmarks, and a lack of end-to-end training that integrates verified feedback. To bridge these gaps, the authors introduce SLAI T-Rex, a full-stack framework that couples system-level optimization on the Ascend SuperPOD with a solver-grounded continued pre-training and supervised fine-tuning pipeline. Their work demonstrates both the viability of large-scale MoE training on Ascend NPUs and significant accuracy gains in structured OR reasoning through targeted domain adaptation.

Dataset

The authors construct two main data streams for post-training an OR-capable model: a continued pre-training (CPT) corpus and a supervised fine-tuning (SFT) dataset refined through self-distillation and AI-assisted quality gates.

CPT Corpus: OR-CPT Data Construction and Domain-Dominant Mixture

• Synthetic OR data comes from the OR-CPT Data Engine, a solver-verified bidirectional synthesis workflow. Parameterized generators span 11+ families (assignment, scheduling, facility location, network flow, production planning, lot sizing, routing, covering, transportation, portfolio optimization, revenue management). Each instance is solved with Gurobi to record solver status, objective, variable assignments, and constraint diagnostics. Instance-level filters remove infeasible, degenerate all-zero, duplicate parameter/configurations, and samples with weak optimisation tradeoffs. Surviving instances are rendered into business-oriented natural-language problems (scenario catalog, no answer leakage) and then forwarded to a model that reconstructs variables, constraints, objective, and Gurobi code. The reconstruction is checked against the solver reference: a sample is accepted only when the Gurobi program reaches a valid status and the objective matches the reference.
• Collected OR resources include OptMATH, ReSocratic, OptiBench, Text2Opt-Bench, OR-Instruct, NLP4LP, ORQA-derived data, MAMO, OR research papers, Gurobi modeling examples, and textbook-style materials. All are converted to a unified CPT format, cleaned, and deduplicated.
• Scale: approximately 100K samples serve as the OR data pool.
• Mixture for CPT: a domain-dominant blend where OR documents provide the primary adaptation signal.
  • Mathematical component: selected AutoMathText-V2 subsets (symbolic formulation, derivation, multi-step reasoning).
  • Code component: NVIDIA Nemotron pretraining data (scientific coding from Nemotron Nano series, code-concept data from Nemotron 3 Super).
  • General English: Ultra-FineWeb-L3, derived from Ultra-FineWeb.
  • The exact sampling ratios are treated as recipe-level hyperparameters.

SFT Dataset: Self-Distilled OR Modeling Data Flywheel

• The input is a set of seed samples — original problems, reference answers, code, data files, or LP files from high-quality modeling QA tasks — used only as structural references.
• Three intermediate representations (IRs) control the synthesis:
  • L1 Canonical IR: normalizes heterogeneous seeds, records source path, target contract, data interface, risk labels.
  • L2 Semantic IR: extracts mathematical-programming semantics (objective, sets, parameters, variables, constraints, data interface).
  • L3 Synthetic IR: generates a new modeling-task abstraction that varies domain, entities, objective semantics, variable/constraint families, parameter schema, numerical range, and data shape, while avoiding any exact copying of seed content.
• Leakage controls compare candidate samples against held-out evaluation sets at surface-text and modeling-structure levels; semantic-similarity filtering rejects near-duplicates, and IR-level constraints prevent reusing the same objective semantics, constraint patterns, parameter schema, or numerical values.
• Synthetic IRs are rendered into three problem formats: DP (data embedded in text), DT (structured Markdown tables), DPS (external data files such as JSON/CSV). Answers follow a markup with `

Method

Training trillion-parameter Mixture-of-Experts (MoE) models like DeepSeek-V4-Pro on Ascend SuperPOD introduces complex workloads due to expert routing and sparse attention. The authors first characterize the end-to-end training bottlenecks by decomposing the latency of a training step into four major components: useful computation, non-overlapped communication, pipeline bubbles, and NPU idle time caused by synchronization and load imbalance.

Profiling results indicate that collecting partial results from Tensor Parallelism (TP) and Data Parallelism (DP) partitions, along with send and receive latency from traditional pipeline parallelism, constitute the primary latency bottlenecks. Furthermore, architecture-specific kernels such as sparse attention dominate computation time because they are memory- and control-bound rather than compute-bound.

To address these inefficiencies, the authors propose a hierarchical three-layer optimization strategy to improve end-to-end training performance.

This framework encompasses optimal computation-communication orchestration via optimized training parallelism, agent-driven optimization of top-ranked bottleneck kernels using AuraKernel, and full-stack operator fusion to eliminate fragmented kernel overhead.

Regarding the parallel execution strategy, the authors enable Expert-Tensor Parallelism (ETP) for MoE layers to reduce the exposed communication overhead introduced by TP. This approach decouples the parallelism choices between attention and MoE modules. Attention layers adopt a lower TP degree to improve GEMM efficiency on large dense matrices, while MoE layers use ETP to preserve the full expert width.

As illustrated in the figure, the ETP token dispatcher structures the MoE layer such that dispatch and combine wrap the expert grouped-GEMM with an EP all-to-all and an ETP all-gather or reduce-scatter. With MoE Parallel Folding, ETP folds into EP as a single group, effectively eliminating separate ETP collectives. Additionally, the authors adopt Virtual Pipeline Parallelism (VPP) to mitigate pipeline bubbles while keeping the per-device memory footprint within the hardware budget.

For communication scheduling, the MoE module implements token dispatch and combine using asynchronous all-to-all collectives, deferring synchronization until the communicated results are consumed.

The authors further optimize this by hiding the synchronization stall preceding the dispatch operation. Because the shared expert depends only on the MoE-layer input and is independent of the routing-derived split sizes, its forward computation is advanced to overlap with the synchronization interval. This scheduling strategy fills an otherwise idle period with useful computation, effectively converting barrier-induced latency into productive execution.

To tackle the memory- and control-bound bottleneck kernels, the authors propose AuraKernel, an end-to-end AscendC kernel optimization agent.

AuraKernel operates in three stages. First, it uses Operations Research to formulate operator tuning as a constrained performance-modeling problem, generating an optimized tiling strategy tailored to the target computation shape. Second, it enters a hardware-grounded iterative optimization loop where a harness compiles and profiles candidates, and K-Search explores a branchable tree of optimization directions based on profiling feedback. Finally, successful indicator-strategy-code patterns are distilled into reusable skills, allowing the agent to systematically restructure large kernels and avoid the suboptimal efficiency of purely heuristic optimization.

Experiment

The evaluation first analyzes the original model's misalignment between solver executability and structural equivalence, revealing that sampling and prompting reduce surface errors but leave structural modeling failures. Operator-level kernel optimization and fusion reduce global memory traffic and launch overhead, while post-training experiments show that SFT with cleaning and chain-of-thought enhancement recovers natural-language optimization ability, and continued pre-training (CPT) initialization yields consistent gains on structural equivalence and executability. The full CPT+SFT pipeline demonstrates complementarity: SFT provides protocol and format alignment, while CPT supplies domain-structural priors, together improving all OR benchmarks without sacrificing general capabilities when a balanced data mixture is used.

Communication latency consumes 52.2% of total device kernel time, with tensor/data and pipeline parallelism dominating. Tensor/data parallelism alone accounts for 25.2% of total kernel time, pipeline parallelism 19.2%, while expert parallelism contributes a modest 7.6%. Tensor/data parallelism and pipeline parallelism together account for 85.1% of communication time (44.4% of total kernel time), making intra-layer reductions and inter-stage point-to-point transfers the primary bottlenecks. Expert parallelism adds only 14.5% to communication time (7.6% of total kernel time), indicating that the AllToAllV token dispatch and combine incurs minimal overhead on this interconnect.

Profiling of DeepSeek-V4-Pro training on Ascend 910C reveals a heavy head of architecture-intrinsic kernels, with sparse attention gradient computation alone occupying 16.86% of total kernel time, while standard MatMul and grouped MoE expert GEMMs achieve high compute utilization. Many top contributors, including element-wise operations and the dominant sparse attention kernel, are constrained by data movement rather than arithmetic throughput, motivating both kernel tuning and fusion of launch-fragmented operator chains. The largest single kernel, SparseAttnSharedkvGrad, takes 16.86% of device compute duration but shows only 9.5% MAC utilization and 15.1% cube utilization, indicating that its latency is dominated by memory access rather than computation. MatMulV3 and GroupedMatmul (MoE experts) are compute-intensive, with MAC utilizations of 52.5% and 86.8% and cube utilizations of 85.4% and 95.4%, respectively, while their MTE2 load utilization exceeds 96%. Element-wise operators Cast and Add together consume over 11% of task duration, but their low vector compute utilization (5.7% and 14.0%) and high MTE2 load utilization (89.7% and 93.4%) classify them as memory-bound, launch-fragmented kernels.

The mHC fusion suite compresses multi-operator eager chains and generic Triton kernels into five hardware-optimized AscendC kernels. Each kernel preserves a precise mathematical contract (row-wise reduction, weighted expansion, or post-BMM addition) while keeping intermediates in Unified Buffer, emitting full output tensors to avoid scatter writebacks, and following the numerical contract for mixed-precision boundaries. wind_rms_norm_without_weight replaces a chain of power, reduction, addition, and reciprocal square root with a single row-wise reduction that produces an FP32 reciprocal-scale output from local dataflow. WindHcPreBmmForward fuses casts and a weighted-expansion contraction into a tiled kernel that broadcasts FP32 weights, takes BF16 inputs, and writes BF16 output. WindMhcPostPart merges casts, an add, and a post-BMM1 operation into a kernel that tiles over rows and feature dimensions, keeping the expansion in Unified Buffer and outputting BF16 directly. Both backward kernels compute all gradients within one tiled pass: wind_rms_norm_without_weight_backward computes a row scalar once and expands it across the dimension, while WindHcPreBmmBackward performs weighted expansion and reduction to produce input and weight gradients simultaneously.

The original DeepSeek-V4-Flash checkpoint exhibits a wide gap between solver executability and structural correctness: B4O-Feasible reaches 60.47% at Pass@1 while B4O-ORGEval only hits 34.26%, even though the code/build pass rate is 79.19%. Increasing sampling (Pass@16) lifts solver-value benchmarks—B4O-Feasible rises to 80.52% and NL4OPT to 93.77%—but B4O-ORGEval only improves to 60.66%, still below the 71.57% achieved with 5-shot prompting, indicating that prompting activates latent formulation knowledge that sampling alone cannot reliably select. Under Pass@1, B4O-Feasible scores 60.47% while B4O-ORGEval reaches only 34.26%, and the 79.19% code/build pass rate shows that many executable solutions fail to preserve the correct optimization structure. Pass@16 boosts B4O-Feasible to 80.52% and NL4OPT to 93.77%, but B4O-ORGEval peaks at 60.66%, far from the 71.57% obtained with 5-shot prompting, proving that sampling alone does not reliably recover canonical formulations.

Across all settings, structural equivalence mismatches form the largest share of failures, while protocol compliance errors dominate at zero-shot but shrink dramatically with added examples. Few-shot prompting shifts the error mix toward ratio and nonlinear modeling issues and away from surface-level formatting problems. Structural equivalence errors are the leading failure family overall (28.3% share) and become even more prominent at Pass@16 (34.3%). Protocol, API, and schema errors peak in the Pass@1 setting (30.1%) but drop sharply to 13.9% in the 5-shot setting. Ratio and nonlinear modeling errors more than double from Pass@1 (11.6%) to the 5-shot regime (17.9%). Discrete-variable semantics errors fall at Pass@16 (8.6%) but rise again to 15.1% under 5-shot prompting.

Profiling of DeepSeek-V4-Pro training on Ascend 910C shows that tensor/data and pipeline parallelism dominate communication time, while expert parallelism adds minimal overhead, and many compute kernels are memory-bound rather than compute-limited. A suite of fused kernels reduces data movement and launch overhead by keeping intermediates on-chip and preserving precise mathematical contracts. Evaluation of the DeepSeek-V4-Flash checkpoint on optimization benchmarks reveals a large gap between generating executable code and producing structurally correct formulations, where few-shot prompting activates latent knowledge that sampling alone cannot reliably recover, and error analysis shows that structural equivalence mismatches persist as the dominant failure mode even under increased sampling.