MindSpore Backend

Support Overview

MindSpeed LLM now supports integration with Huawei's in-house AI framework MindSpore. It aims to provide an easy-to-use end-to-end large language model training solution across Huawei's full stack and deliver a more extreme performance experience. The MindSpore backend provides a set of APIs that align with PyTorch. Therefore, you can switch seamlessly without additional code adaptation.

---

News

🚀🚀🚀The MindSpore backend now supports DeepSeek-V3/Qwen3/GLM-4.5! 🚀🚀🚀

Version Compatibility Table

The dependency compatibility for MindSpeed LLM plus the MindSpore backend is listed below. For installation steps, see MindSpeed LLM Installation Guide.

Dependency	Version
Ascend NPU driver	In development
Ascend NPU firmware
Toolkit	CANN 8.5.0
Kernel
Ascend Transformer Boost acceleration library (NNAL)
Python	3.10
MindSpore	2.8.0

Note: The master branch uses development versions of the driver and CANN package. Therefore, some new features on master may not be supported by older dependency versions. To use a stable version, switch to the commercial release branch and install the corresponding dependency versions.

Model Support

The MindSpore backend only supports models implemented in mcore. The current model support details are listed below, and support for more models will be added gradually.

Model Category	Model List
Supported Models	supported_models

Feature Support

The following table shows support for the key acceleration features of MindSpeed on the MindSpore backend. Some unsupported features will be added in later iterations. Stay tuned.

Scenario	Feature	Support
SPTD parallelism	Tensor parallelism	✅
	Pipeline parallelism	✅
	Virtual pipeline parallelism	✅
	Sequence parallelism	✅
	Noop Layers	✅
	Fully sharded parallelism	Does not currently support enabling `pp` or the `--reuse-fp32-param` parameter.
Long-sequence parallelism	Ascend Ring Attention long-sequence parallelism	✅
	Ulysses long-sequence parallelism	✅
MOE	MOE expert parallelism	✅
	MOE dispatch communication optimization	Only `alltoall` is supported.
Memory optimization	Parameter copy reuse	Must be used together with the distributed optimizer feature.
	Distributed optimizer	✅
	Recomputation	✅
	Norm recomputation	✅
	Virtual Optimizer	✅
Fused operators	Flash attention	✅
	Variable-length flash attention	✅
	Fused RMSNorm	✅
	Fused SwiGLU	✅
	Fused rotary position embedding	✅
	GMM	✅
	Matmul Add	✅
Communication optimization	Gradient reduce communication overlap	✅
	Weight all-gather communication overlap	✅
	CoC	✅

Online Inference

Feature	Supported
Streaming inference	✅
Chat conversation	✅
YARN context extension	✅

Open Dataset Evaluation

Coming soon. Stay tuned.

Developer Toolchain

Data Preprocessing

The MindSpore backend fully supports data preprocessing for MindSpeed LLM pretraining, supervised fine-tuning, RLHF, and other tasks.

Scenario	Dataset	MCore	Released	Contributor
Pretraining	Pretraining data processing	✅	✅	【Ascend】
Fine-tuning	Alpaca style	✅	✅
	ShareGPT style	✅	✅
DPO	Pairwise data processing	✅	✅	【NAIE】

Weight Conversion

The weight conversion for the MindSpeed MindSpore backend is consistent with the PyTorch backend. It currently supports mutual conversion between Hugging Face and Megatron-Core weight formats. For the weight conversion parameters and usage, see Weight Conversion.

Source Format	Target Format	Sharding Features	LoRA	Contributor
huggingface	megatron-core	tp, pp, dpp, vpp, cp, ep, loop layer	❌	【Ascend】
megatron-core	huggingface		✅
	megatron-core	tp, pp, dpp, vpp, cp, ep, loop layer	❌

Performance Profiling

Scenario	Feature	MCore	Contributor
Performance profiling	Collect profiling data based on Ascend chips	✅	【Ascend】

High Availability

Scenario	Feature	MCore	Contributor
High availability	Enable deterministic computation based on Ascend chips	✅	【Ascend】