HyperAIHyperAI

Command Palette

Search for a command to run...

De TeleChat à Xing 4 : faire progresser les modèles agentiques de pointe sur puces et frameworks nationaux

Résumé

Nous présentons Xing 4, une série de grands modèles de langage qui prolonge la lignée TeleChat grâce à des avancées coordonnées en architecture de modèle et en infrastructure d’entraînement sur Ascend. Alors que la plupart des modèles open source sont entraînés sur GPU, le passage à l’échelle sur puces et frameworks nationaux pose des défis systèmes importants. Ce rapport décrit les principes de conception et l’ingénierie système sous-jacents à la série, couvrant le choix d’architecture, le pré-entraînement à grande échelle, ainsi que le post-entraînement au raisonnement et aux capacités agentiques. Nous étudions comment l’activation parcimonieuse, la conception de l’attention, la profondeur et la largeur du modèle, ainsi que le décodage spéculatif équilibrent la capacité du modèle avec le calcul, l’accès mémoire et les surcoûts d’exécution. Des configurations représentatives combinent MLA avec DSA et IndexShare inter-couches pour le service à long contexte, intègrent des opérateurs MHC optimisés pour améliorer la stabilité et l’efficacité computationnelle, adoptent une topologie de mélange d’experts (MoE) plus large et moins profonde, et associent un modèle de brouillon par blocs à des têtes de correction légères pour le décodage spéculatif. Nous avons ouvert le code d’un membre MoE de 29B, un modèle entièrement entraîné sur NPU Ascend avec MindSpore, n’activant que 4B de paramètres par jeton tout en prenant en charge nativement des contextes de 256K jetons. Notre infrastructure de pré-entraînement s’appuie sur MindSpore HyperParallel, combinant un partitionnement hybride tenant compte de la topologie, une exécution efficace d’optimiseurs distribués et une gestion coordonnée des charges de travail à longues séquences et de la mémoire d’activation. L’affinité CPU et l’isolation des cœurs réduisent en outre les interférences d’ordonnancement de l’hôte. Ensemble, ces mécanismes soutiennent le pré-entraînement d’un modèle de 285B de paramètres sur 8 192 cartes Ascend 910C issues d’un cluster de production à grande échelle, offrant un cadre concret pour les études architecturales et systèmes. Les mesures montrent jusqu’à 4,68× de débit de préremplissage par rapport à une base MLA seule, 2,67× à 3,77× de débit de décodage grâce au décodage spéculatif, et une amélioration de débit de 16,1 % grâce à l’élimination du parallélisme pipeline. Pour le post-entraînement, nous introduisons TeleGym, une plateforme unifiée d’exécution d’agents qui relie l’exécution, l’évaluation et la construction de trajectoires à l’optimisation de politique. L’exécution asynchrone désagrégée, le suréchantillonnage borné et le déploiement partiel s’adaptent à des durées de tâches hétérogènes tout en préservant la continuité des trajectoires entre les mises à jour de politique. Ensemble, ces avancées établissent une base d’entraînement orientée Ascend pour le développement continu de Xing 4 à différentes échelles de modèles et pour des charges de travail de raisonnement et d’agents de plus en plus exigeantes. Le membre 29B en open source et l’entraînement en cours à mille milliards de paramètres prolongent encore la série au sein de l’écosystème Ascend et MindSpore.

One-sentence Summary

Researchers from Xingchen AGI Lab, China Telecom, Huawei, and Harbin Institute of Technology (Shenzhen) introduce Xing 4, a TeleChat-derived agentic model series trained on Ascend NPUs and MindSpore that combines MLA, DSA, cross-layer IndexShare, optimized MHC operators, a wider-and-shallower mixture-of-experts topology, and block-wise speculative decoding with lightweight correction heads; the open-sourced 29B MoE member activates only 4B parameters per token, supports native 256K-token contexts, and achieves up to 4.68× prefill throughput, 2.67× to 3.77× decoding throughput from speculative decoding, and a 16.1% throughput improvement from eliminating pipeline parallelism, while TeleGym supports agent post-training through disaggregated asynchronous execution, bounded oversampling, and partial rollout.

Key Contributions

  • The paper contributes a coordinated architecture and decoding design for the Xing 4 model series, combining MLA with DSA and cross-layer IndexShare for long-context serving, optimized MHC operators, a wider-and-shallower MoE topology, and a block-wise draft model with lightweight correction heads for speculative decoding. An open-sourced 29B MoE member trained entirely on Ascend NPUs activates 4B parameters per token with native 256K-token contexts, and measurements show up to 4.68× prefill throughput over an MLA-only baseline, 2.67× to 3.77× decoding throughput from speculative decoding, and a 16.1% throughput improvement from eliminating pipeline parallelism.
  • This work contributes an Ascend-oriented large-scale pretraining system built on MindSpore HyperParallel, integrating topology-aware hybrid sharding, owner-based Muon execution, long-sequence workload balancing, KV compaction, activation-memory management, and CPU affinity and core isolation. The system is demonstrated by pretraining a 285B-parameter MoE model on 8,192 Ascend 910C cards.
  • This work presents TeleGym, a unified agent execution platform that connects execution, evaluation, and trajectory construction to policy optimization, using disaggregated asynchronous execution, bounded oversampling, and partial rollout to accommodate heterogeneous task durations while preserving trajectory continuity across policy updates.

Introduction

The Xing 4 series extends the TeleChat family of large language models, building on prior pretraining, instruction tuning, reasoning, and mixture-of-experts work while targeting long-context reasoning, code generation, and agentic applications on the Ascend and MindSpore ecosystem. Scaling such models introduces practical bottlenecks: larger capacity must be balanced against per-token computation and the cost of storing, communicating, and updating the full model, while long-context attention adds indexing, KV-access, and activation-memory costs; packed documents can also create uneven computational loads, and host scheduling or variable task durations can stall training and reinforcement learning. The authors address these challenges through Ascend-oriented architectural design that jointly considers sparse activation, compressed KV attention, depth-width tradeoffs, and speculative decoding; a pretraining system with topology-aware sharding, owner-based Muon optimization, long-sequence workload balancing, and CPU scheduling controls; and TeleGym, a unified asynchronous reinforcement-learning platform for reasoning, coding, and multi-turn agent tasks.

Dataset

The authors describe a synthesized task dataset for post-training general and code agents. It is not drawn from a static corpus. Instead, tasks are proposed, refined, validated, and deduplicated through multiple agent harnesses, with execution handled through the TeleGym runtime.

  • Dataset composition and sources

    • General-agent tasks are built around information retrieval, document processing, and tool use.
    • Code-agent tasks are built around software projects and programming requirements, including implementation and bug fixing.
    • Multiple harnesses contribute to task design, instruction refinement, test review, and validation.
  • General-agent subset

    • Instructions are paired with relevant files and tool interfaces.
    • Covers single-turn and multi-turn scenarios with varied constraints and interaction steps.
    • Feedback from multiple harnesses is used to refine instructions, tool requirements, and evaluation criteria.
    • Validation uses programmatic checks or model-based assessment against shared criteria.
  • Code-agent subset

    • Tasks start from an initial code workspace and dependency configuration.
    • Shared executable tests verify both the requested changes and preservation of existing functionality.
    • Independent attempts expose environment issues and mismatches between task descriptions and tests before selection.
  • Processing and filtering

    • Each task combines an instruction, an initial environment, and evaluation criteria.
    • Candidate tasks are validated independently across harnesses from equivalent task conditions.
    • Validation and deduplication remove unusable or repetitive examples.
    • Accepted tasks enter the shared rollout pipeline through TeleGym’s execution interface.
  • Usage in the model

    • The accepted tasks are used as executable post-training tasks for general and code agents in the shared rollout pipeline.
    • The text does not specify dataset sizes, training split proportions, or mixture ratios.
    • No cropping strategy or explicit metadata schema is described beyond the task structure of instruction, initial environment, and evaluation criteria.

Method

The authors design the Xing 4 architecture to balance model capacity, long-context execution, and hardware efficiency on Ascend 910C clusters. As a concrete implementation, the 285B representative model utilizes a decoder-only Mixture-of-Experts (MoE) backbone. Refer to the framework diagram for the structural layout. The model consists of 45 transformer layers, where the first three layers use Dense Feed-Forward Networks (FFNs) and the remaining 42 layers employ MoE FFNs to expand capacity through sparse activation. To address long-context inference bottlenecks, the attention mechanism combines Multi-head Latent Attention (MLA) with DeepSeek Sparse Attention (DSA) and IndexShare. MLA compresses KV representations to reduce cache memory, while DSA uses a lightning indexer to select Top-K historical KV states. IndexShare further reuses sparse indices across layers to minimize repeated indexer computation. For inference-time efficiency, the architecture integrates DSpark, a speculative decoding component that uses a block-wise draft model, a lightweight Markov head, and an acceptance confidence head to increase the number of tokens committed per verifier invocation.

To support large-scale pretraining without the structural pipeline bubbles inherent in pipeline parallelism, the authors implement HyperParallel-FSDP, a hybrid sharded data parallelism scheme tailored to the Ascend SuperPoD architecture. Model parameters are FSDP-sharded within each SuperPoD and replicated across SuperPoDs, confining frequent weight-gathering and sharded-gradient synchronization to high-bandwidth intra-SuperPoD links. The backward-pass workflow is illustrated in the following figure.

To optimize communication, HyperParallel-FSDP eliminates the flattened intermediate buffers and associated copy-in and copy-out operations used in standard FSDP. Instead, it performs AllGather and ReduceScatter directly on individual parameter tensors, achieving zero-copy communication as shown in the comparison below.

Furthermore, the system implements a three-way communication overlap strategy during the backward pass. Intra-SuperPoD AllGather and ReduceScatter operations are asynchronously scheduled and overlapped with backward computation, while inter-SuperPoD gradient reduction is deferred and fused. The execution timelines demonstrate how this optimized schedule hides communication latency behind layer computations.

For the distributed Muon optimizer, which requires full weight matrices for Newton-Schulz iteration, the authors introduce an owner-based execution strategy to remove redundant full-matrix computations across ranks. In the baseline approach, every rank reconstructs the full matrix and runs the iteration independently. The optimized approach assigns a single owner rank within each shard group to perform the full-matrix computation. Non-owner ranks send their input shards to the owner via point-to-point transfers. The owner assembles the input, runs the Newton-Schulz iteration, and scatters the scaled update shards back. The reassignment of full-matrix work is depicted in the figure below.

Experiment

The experiments cover speculative decoding, distributed training, gradient scheduling, and host-resource management. DSpark is benchmarked against the base configuration and MTP3 on mathematical reasoning and code generation tasks, demonstrating improved decoding throughput through parallel block-wise drafting and lightweight Markov corrections. HyperParallel-FSDP is evaluated on a 285B-parameter MoE model on a 1,024-NPU cluster and compared with pipeline-parallel schedules, showing that hybrid sharding eliminates the pipeline bubble and improves step time. The layer-wise indexer backward optimization is tested against an HSDP plus Muon baseline and provides the largest single-feature gain by avoiding late auxiliary-graph backpropagation, while CPU pinning and core isolation are evaluated on 512- and 8,192-accelerator clusters and show that reducing host scheduling contention increases training throughput, with greater benefits at larger scale.

Enabling CPU pinning and core isolation together increases training throughput relative to the baseline at both tested cluster sizes. The improvement is larger on the 8,192-accelerator cluster than on the 512-accelerator cluster. The results link the larger gain to reduced scheduling delays and shorter critical-path synchronization waits. Combined CPU pinning and core isolation improved throughput at both cluster scales. The larger cluster showed a greater relative gain, attributed to reduced scheduling delays and shorter synchronization waits.

In the 285B simulation, adding DSA to MLA improves latency and throughput relative to the MLA baseline while reducing KV HBM access. Combining DSA with cross-layer index sharing yields the largest gains, with the lowest time-to-first-token and the highest prefill and decode throughput among the variants. All metrics are normalized to the MLA baseline. MLA + DSA reduces time-to-first-token to about a third of the MLA baseline and more than doubles prefill throughput. With cross-layer index sharing, the variant achieves the lowest KV HBM access and time-to-first-token while delivering the highest prefill and decode throughput.

The depth-width study compares a baseline configuration with a wider-shallower variant. The wider-shallower model reduces the number of layers but increases expert intermediate size and the number of experts used per token. Hidden size, attention heads, routed expert count, and shared expert count remain unchanged between the two configurations. The wider-shallower configuration lowers depth while increasing expert capacity per token and expert intermediate size. Hidden size, attention heads, routed expert count, and shared expert count stay fixed across both configurations.

In the 285B depth-width comparison, the selected wider-shallower configuration improves normalized inference efficiency relative to the baseline. It lowers latency while increasing throughput, which supports its selection over the reference design. The wider-shallower configuration reduces relative time-to-first-token and per-output-token latency compared with the baseline. It achieves higher prefill throughput and decode throughput than the reference configuration.

Across six mathematical-reasoning and code-generation benchmarks, DSpark consistently achieves higher decoding throughput than both the base configuration and MTP3 for the 285B model. MTP3 improves output throughput by roughly 1.8x to 1.9x over the base, while DSpark reaches about 2.7x to 3.8x over the base and about 1.5x to 2.1x over MTP3. The largest DSpark advantage appears on HumanEval, and the smallest appears on LiveCodeBench. DSpark delivers the highest output throughput on every benchmark, with speedups over the base configuration ranging from roughly 2.7x to 3.8x. Relative to MTP3, DSpark adds the largest throughput gain on HumanEval and the smallest gain on LiveCodeBench.

Across the experiments, the work validates system and model optimizations: CPU pinning combined with core isolation improves training throughput at both cluster scales, with larger gains attributed to reduced scheduling and synchronization delays, while adding DSA to MLA and combining it with cross-layer index sharing lowers KV HBM access and improves prefill and decode throughput. A wider-shallower 285B configuration also lowers latency and increases throughput relative to the deeper baseline, supporting the selected depth-width design. On mathematical reasoning and code generation benchmarks, DSpark consistently achieves the highest decoding throughput, outperforming both the base configuration and MTP3.


Créer de l'IA avec l'IA

De l'idée au lancement — accélérez votre développement IA avec le co-codage IA gratuit, un environnement prêt à l'emploi et le meilleur prix pour les GPU.

Codage assisté par IA
GPU prêts à l’emploi
Tarifs les plus avantageux

HyperAI Newsletters

Abonnez-vous à nos dernières mises à jour
Nous vous enverrons les dernières mises à jour de la semaine dans votre boîte de réception à neuf heures chaque lundi matin
Propulsé par MailChimp