Command Palette
Search for a command to run...
TeleChat에서 Xing 4로: 국산 칩과 프레임워크 기반 최전선 에이전트 모델의 진전
TeleChat에서 Xing 4로: 국산 칩과 프레임워크 기반 최전선 에이전트 모델의 진전
초록
본 논문은 Ascend 환경에서 모델 아키텍처와 학습 인프라의 유기적 개선을 통해 TeleChat 계보를 확장한 대규모 언어 모델 시리즈인 Xing 4를 제시한다. 대부분의 오픈소스 모델이 GPU에서 학습되는 반면, 국산 칩과 프레임워크에서의 확장은 중대한 시스템적 도전 과제를 수반한다. 본 보고서는 아키텍처 선정, 대규모 사전학습, 추론 및 에이전트 사후학습에 걸친 본 시리즈의 설계 원칙과 시스템 엔지니어링을 기술한다. 희소 활성화, 어텐션 설계, 모델 깊이와 폭, 추측적 디코딩이 모델 용량과 연산량, 메모리 접근, 실행 오버헤드 사이의 균형을 어떻게 조율하는지 연구한다. 대표 구성은 장문 컨텍스트 서빙을 위해 MLA와 DSA 및 계층 간 IndexShare를 결합하고, 안정성과 계산 효율을 높이기 위해 최적화된 MHC 연산자를 통합하며, 더 넓고 얕은 전문가 혼합(MoE) 토폴로지를 채택하고, 추측적 디코딩을 위해 블록 단위 초안 모델과 경량 교정 헤드를 결합한다. 우리는 MindSpore를 사용하여 Ascend NPU에서 전적으로 학습된 29B MoE 모델을 오픈소스로 공개했으며, 이 모델은 토큰당 4B 파라미터만 활성화하면서 네이티브 256K 토큰 컨텍스트를 지원한다. 사전학습 인프라는 MindSpore HyperParallel을 기반으로 하며, 토폴로지 인지형 하이브리드 샤딩, 효율적인 분산 옵티마이저 실행, 장문 시퀀스 워크로드와 활성화 메모리의 통합 관리를 결합한다. CPU 친화도와 코어 격리는 호스트 스케줄링 간섭을 추가로 줄인다. 이러한 메커니즘은 대규모 프로덕션 클러스터에서 확보한 8,192개의 Ascend 910C 카드에서 285B 파라미터 모델의 사전학습을 지원하여 아키텍처 및 시스템 연구를 위한 구체적 환경을 제공한다. 측정 결과 MLA 단독 기준선 대비 최대 4.68배의 프리필 처리량, 추측적 디코딩으로 2.67배~3.77배의 디코딩 처리량, 파이프라인 병렬화 제거로 16.1%의 처리량 향상을 확인했다. 사후학습을 위해 실행, 평가, 궤적 구축을 정책 최적화와 연결하는 통합 에이전트 실행 플랫폼인 TeleGym을 소개한다. 분리형 비동기 실행, 유계 오버샘플링, 부분 롤아웃은 정책 업데이트 전반에 걸쳐 궤적 연속성을 유지하면서 이질적인 작업 소요 시간을 수용한다. 이러한 진전은 모델 규모와 점점 더 까다로워지는 추론 및 에이전트 워크로드 전반에 걸쳐 Xing 4의 지속적 발전을 위한 Ascend 중심의 학습 기반을 확립한다. 오픈소스로 공개된 29B 모델과 진행 중인 조 단위 파라미터 학습은 Ascend 및 MindSpore 생태계 내에서 본 시리즈를 더욱 확장한다.
One-sentence Summary
Researchers from Xingchen AGI Lab, China Telecom, Huawei, and Harbin Institute of Technology (Shenzhen) introduce Xing 4, a TeleChat-derived agentic model series trained on Ascend NPUs and MindSpore that combines MLA, DSA, cross-layer IndexShare, optimized MHC operators, a wider-and-shallower mixture-of-experts topology, and block-wise speculative decoding with lightweight correction heads; the open-sourced 29B MoE member activates only 4B parameters per token, supports native 256K-token contexts, and achieves up to 4.68× prefill throughput, 2.67× to 3.77× decoding throughput from speculative decoding, and a 16.1% throughput improvement from eliminating pipeline parallelism, while TeleGym supports agent post-training through disaggregated asynchronous execution, bounded oversampling, and partial rollout.
Key Contributions
- The paper contributes a coordinated architecture and decoding design for the Xing 4 model series, combining MLA with DSA and cross-layer IndexShare for long-context serving, optimized MHC operators, a wider-and-shallower MoE topology, and a block-wise draft model with lightweight correction heads for speculative decoding. An open-sourced 29B MoE member trained entirely on Ascend NPUs activates 4B parameters per token with native 256K-token contexts, and measurements show up to 4.68× prefill throughput over an MLA-only baseline, 2.67× to 3.77× decoding throughput from speculative decoding, and a 16.1% throughput improvement from eliminating pipeline parallelism.
- This work contributes an Ascend-oriented large-scale pretraining system built on MindSpore HyperParallel, integrating topology-aware hybrid sharding, owner-based Muon execution, long-sequence workload balancing, KV compaction, activation-memory management, and CPU affinity and core isolation. The system is demonstrated by pretraining a 285B-parameter MoE model on 8,192 Ascend 910C cards.
- This work presents TeleGym, a unified agent execution platform that connects execution, evaluation, and trajectory construction to policy optimization, using disaggregated asynchronous execution, bounded oversampling, and partial rollout to accommodate heterogeneous task durations while preserving trajectory continuity across policy updates.
Introduction
The Xing 4 series extends the TeleChat family of large language models, building on prior pretraining, instruction tuning, reasoning, and mixture-of-experts work while targeting long-context reasoning, code generation, and agentic applications on the Ascend and MindSpore ecosystem. Scaling such models introduces practical bottlenecks: larger capacity must be balanced against per-token computation and the cost of storing, communicating, and updating the full model, while long-context attention adds indexing, KV-access, and activation-memory costs; packed documents can also create uneven computational loads, and host scheduling or variable task durations can stall training and reinforcement learning. The authors address these challenges through Ascend-oriented architectural design that jointly considers sparse activation, compressed KV attention, depth-width tradeoffs, and speculative decoding; a pretraining system with topology-aware sharding, owner-based Muon optimization, long-sequence workload balancing, and CPU scheduling controls; and TeleGym, a unified asynchronous reinforcement-learning platform for reasoning, coding, and multi-turn agent tasks.
Dataset
The authors describe a synthesized task dataset for post-training general and code agents. It is not drawn from a static corpus. Instead, tasks are proposed, refined, validated, and deduplicated through multiple agent harnesses, with execution handled through the TeleGym runtime.
-
Dataset composition and sources
- General-agent tasks are built around information retrieval, document processing, and tool use.
- Code-agent tasks are built around software projects and programming requirements, including implementation and bug fixing.
- Multiple harnesses contribute to task design, instruction refinement, test review, and validation.
-
General-agent subset
- Instructions are paired with relevant files and tool interfaces.
- Covers single-turn and multi-turn scenarios with varied constraints and interaction steps.
- Feedback from multiple harnesses is used to refine instructions, tool requirements, and evaluation criteria.
- Validation uses programmatic checks or model-based assessment against shared criteria.
-
Code-agent subset
- Tasks start from an initial code workspace and dependency configuration.
- Shared executable tests verify both the requested changes and preservation of existing functionality.
- Independent attempts expose environment issues and mismatches between task descriptions and tests before selection.
-
Processing and filtering
- Each task combines an instruction, an initial environment, and evaluation criteria.
- Candidate tasks are validated independently across harnesses from equivalent task conditions.
- Validation and deduplication remove unusable or repetitive examples.
- Accepted tasks enter the shared rollout pipeline through TeleGym’s execution interface.
-
Usage in the model
- The accepted tasks are used as executable post-training tasks for general and code agents in the shared rollout pipeline.
- The text does not specify dataset sizes, training split proportions, or mixture ratios.
- No cropping strategy or explicit metadata schema is described beyond the task structure of instruction, initial environment, and evaluation criteria.
Method
The authors design the Xing 4 architecture to balance model capacity, long-context execution, and hardware efficiency on Ascend 910C clusters. As a concrete implementation, the 285B representative model utilizes a decoder-only Mixture-of-Experts (MoE) backbone. Refer to the framework diagram for the structural layout. The model consists of 45 transformer layers, where the first three layers use Dense Feed-Forward Networks (FFNs) and the remaining 42 layers employ MoE FFNs to expand capacity through sparse activation. To address long-context inference bottlenecks, the attention mechanism combines Multi-head Latent Attention (MLA) with DeepSeek Sparse Attention (DSA) and IndexShare. MLA compresses KV representations to reduce cache memory, while DSA uses a lightning indexer to select Top-K historical KV states. IndexShare further reuses sparse indices across layers to minimize repeated indexer computation. For inference-time efficiency, the architecture integrates DSpark, a speculative decoding component that uses a block-wise draft model, a lightweight Markov head, and an acceptance confidence head to increase the number of tokens committed per verifier invocation.
To support large-scale pretraining without the structural pipeline bubbles inherent in pipeline parallelism, the authors implement HyperParallel-FSDP, a hybrid sharded data parallelism scheme tailored to the Ascend SuperPoD architecture. Model parameters are FSDP-sharded within each SuperPoD and replicated across SuperPoDs, confining frequent weight-gathering and sharded-gradient synchronization to high-bandwidth intra-SuperPoD links. The backward-pass workflow is illustrated in the following figure.
To optimize communication, HyperParallel-FSDP eliminates the flattened intermediate buffers and associated copy-in and copy-out operations used in standard FSDP. Instead, it performs AllGather and ReduceScatter directly on individual parameter tensors, achieving zero-copy communication as shown in the comparison below.
Furthermore, the system implements a three-way communication overlap strategy during the backward pass. Intra-SuperPoD AllGather and ReduceScatter operations are asynchronously scheduled and overlapped with backward computation, while inter-SuperPoD gradient reduction is deferred and fused. The execution timelines demonstrate how this optimized schedule hides communication latency behind layer computations.
For the distributed Muon optimizer, which requires full weight matrices for Newton-Schulz iteration, the authors introduce an owner-based execution strategy to remove redundant full-matrix computations across ranks. In the baseline approach, every rank reconstructs the full matrix and runs the iteration independently. The optimized approach assigns a single owner rank within each shard group to perform the full-matrix computation. Non-owner ranks send their input shards to the owner via point-to-point transfers. The owner assembles the input, runs the Newton-Schulz iteration, and scatters the scaled update shards back. The reassignment of full-matrix work is depicted in the figure below.
Experiment
The experiments cover speculative decoding, distributed training, gradient scheduling, and host-resource management. DSpark is benchmarked against the base configuration and MTP3 on mathematical reasoning and code generation tasks, demonstrating improved decoding throughput through parallel block-wise drafting and lightweight Markov corrections. HyperParallel-FSDP is evaluated on a 285B-parameter MoE model on a 1,024-NPU cluster and compared with pipeline-parallel schedules, showing that hybrid sharding eliminates the pipeline bubble and improves step time. The layer-wise indexer backward optimization is tested against an HSDP plus Muon baseline and provides the largest single-feature gain by avoiding late auxiliary-graph backpropagation, while CPU pinning and core isolation are evaluated on 512- and 8,192-accelerator clusters and show that reducing host scheduling contention increases training throughput, with greater benefits at larger scale.
Enabling CPU pinning and core isolation together increases training throughput relative to the baseline at both tested cluster sizes. The improvement is larger on the 8,192-accelerator cluster than on the 512-accelerator cluster. The results link the larger gain to reduced scheduling delays and shorter critical-path synchronization waits. Combined CPU pinning and core isolation improved throughput at both cluster scales. The larger cluster showed a greater relative gain, attributed to reduced scheduling delays and shorter synchronization waits.
In the 285B simulation, adding DSA to MLA improves latency and throughput relative to the MLA baseline while reducing KV HBM access. Combining DSA with cross-layer index sharing yields the largest gains, with the lowest time-to-first-token and the highest prefill and decode throughput among the variants. All metrics are normalized to the MLA baseline. MLA + DSA reduces time-to-first-token to about a third of the MLA baseline and more than doubles prefill throughput. With cross-layer index sharing, the variant achieves the lowest KV HBM access and time-to-first-token while delivering the highest prefill and decode throughput.
The depth-width study compares a baseline configuration with a wider-shallower variant. The wider-shallower model reduces the number of layers but increases expert intermediate size and the number of experts used per token. Hidden size, attention heads, routed expert count, and shared expert count remain unchanged between the two configurations. The wider-shallower configuration lowers depth while increasing expert capacity per token and expert intermediate size. Hidden size, attention heads, routed expert count, and shared expert count stay fixed across both configurations.
In the 285B depth-width comparison, the selected wider-shallower configuration improves normalized inference efficiency relative to the baseline. It lowers latency while increasing throughput, which supports its selection over the reference design. The wider-shallower configuration reduces relative time-to-first-token and per-output-token latency compared with the baseline. It achieves higher prefill throughput and decode throughput than the reference configuration.
Across six mathematical-reasoning and code-generation benchmarks, DSpark consistently achieves higher decoding throughput than both the base configuration and MTP3 for the 285B model. MTP3 improves output throughput by roughly 1.8x to 1.9x over the base, while DSpark reaches about 2.7x to 3.8x over the base and about 1.5x to 2.1x over MTP3. The largest DSpark advantage appears on HumanEval, and the smallest appears on LiveCodeBench. DSpark delivers the highest output throughput on every benchmark, with speedups over the base configuration ranging from roughly 2.7x to 3.8x. Relative to MTP3, DSpark adds the largest throughput gain on HumanEval and the smallest gain on LiveCodeBench.
Across the experiments, the work validates system and model optimizations: CPU pinning combined with core isolation improves training throughput at both cluster scales, with larger gains attributed to reduced scheduling and synchronization delays, while adding DSA to MLA and combining it with cross-layer index sharing lowers KV HBM access and improves prefill and decode throughput. A wider-shallower 285B configuration also lowers latency and increases throughput relative to the deeper baseline, supporting the selected depth-width design. On mathematical reasoning and code generation benchmarks, DSpark consistently achieves the highest decoding throughput, outperforming both the base configuration and MTP3.