Supported Models for the MindSpore Framework

MindSpore models fall into dense models and sparse models based on how they are implemented. See the support list below for details.

Table field descriptions:

• Model: Model name.
• Download link: Download address for the model weights. Click it to visit repositories such as Hugging Face directly.
• Script location: Training script path for the model in this project. You can use it to quickly locate and run the model.
• Sequence length: Maximum supported text sequence length.
• Training backend: Only models implemented with mcore are supported. Legacy implementations are not planned for support.
• Cluster size: Recommended cluster configuration for model training, in the format "number of nodes × number of devices".
• Supported version: Final supported maintenance version. A blank value means the model is maintained from launch through the current master branch.

Dense Models

Dense models are a traditional deep learning model structure. Their neurons are densely connected, and most or all neurons in each layer connect to all neurons in the next layer. These models are simple and relatively straightforward to train, but they have a large parameter count and higher computational cost.

Model	Download link	Script location	Sequence length	Training backend	Cluster size	Supported version
ChatGLM3	6B	chatglm3	8K	MCore	1x8	2.3.0
			64K	MCore	2x8
GLM4	9B	glm4	8K	MCore	1x8	2.3.0
			32K	MCore	2x8	2.3.0
	32B		8K	MCore	2x8
CodeLlama	34B	codellama	4K	MCore	2x8	2.2.0
InternLM2	20B	internlm2	4K	MCore	1x8	2.2.0
			32K	MCore	1x8
InternLM2.5	1.8B	internlm25	32K	MCore	1x8
	7B		32K	MCore	1x8
	20B		32K	MCore	2x8
Llama-2	7B	llama2	4K	MCore	1x8	2.3.0
			16K	MCore	1x8
			32K	MCore	1x8
	13B		4K	MCore	1x8	2.3.0
	34B		4K	MCore	2x8
	70B		4K	MCore	4x8
Llama-3	8B	llama3	8K	MCore	1x8	2.3.0
	70B		8K	MCore	4x8
Llama-3.1	8B	llama31	8K	MCore	1x8	2.3.0
			128K	MCore	4x8	In progress
	70B		8K	MCore	4x8
Llama-3.2	1B	llama32	8K	MCore	1x8	2.3.0
	3B		8K	MCore	1x8
Llama-3.3	70B-Instruct	llama33	8K	MCore	4x8
Qwen1.5	0.5B	qwen15	8K	MCore	1x8	2.2.0
	1.8B		8K	MCore	1x8
	4B		8K	MCore	1x8
	7B		8K	MCore	1x8
	14B		8K	MCore	1x8
	32B		8K	MCore	4x8
	72B		8K	MCore	8x8
	110B		8K	MCore	8x8
CodeQwen1.5	7B		8K	MCore	1x8	2.2.0
Qwen2	7B	qwen2	4K	MCore	1x8	2.2.0
Qwen2.5	0.5B	qwen25	32K	MCore	1x8
	1.5B		32K	MCore	1x8
	3B		32K	MCore	1x8
	7B		4K	MCore	1x8	2.3.0
			32K	MCore	1x8	2.3.0
	14B		4K	MCore	1x8	2.3.0
			32K	MCore	2x8
	32B		4K	MCore	4x8
			32K	MCore	4x8
	72B		4K	MCore	4x8
			32K	MCore	8x8
Qwen3	0.6B	qwen3	4K	MCore	1x8	2.3.0
	1.7B		4K	MCore	1x8
	4B		4K	MCore	1x8
	8B		4K	MCore	1x8
	14B		4K	MCore	1x8
	32B		4K	MCore	2x8
Yi	34B	yi	4K	MCore	2x8	2.2.0
Yi1.5	6B	yi15	4K	MCore	1x8	2.2.0
	9B		4K	MCore	1x8
	34B		4K	MCore	2x8
Mistral	7B	mistral	32K	MCore	1x8	2.2.0
Gemma	2B	gemma	8K	MCore	1x8	2.2.0
	7B		8K	MCore	1x8
Gemma2	9B	gemma2	8K	MCore	1x8
	27B		8K	MCore	2x8
grok-1	40B	grok-1	8K	MCore	4x8
MiniCPM	2B	minicpm	4K	MCore	1x8	2.2.0
Phi3.5	mini-instruct	phi35	4K	MCore	1x8	2.3.0
DeepSeek-R1-Distill-Qwen	1.5B	deepseek_r1_distill_qwen	4K	MCore	1x8	2.2.0
	7B		4K	MCore	1x8
	14B		4K	MCore	1x8
	32B		8K	MCore	2x8
DeepSeek-R1-Distill-Llama	8B	deepseek_r1_distill_llama	8K	MCore	1x8	2.2.0
	70B		8K	MCore	4x8

Sparse Models

Sparse models use a sparsely connected neuron structure, where only a small number of neurons are connected. A typical sparse model is the Mixture of Experts (MoE) model, which contains multiple expert networks and activates only a subset of experts during each training step. This design can significantly reduce the parameter count and computational complexity, improve training efficiency, and is especially suitable for large datasets and complex tasks. However, sparse model training also has drawbacks. It can easily suffer from unbalanced expert loads, which leads to unstable training.

Model	Download link	Script location	Sequence length	Training backend	Cluster size	Supported version
Qwen3	30B	qwen3_moe	4K	MCore	2x8
	235B		4K	MCore	16x16
Qwen2	57B-A14B	qwen2_moe	4K	MCore	8x8	2.2.0
Mixtral	8x7B	mixtral	32K	MCore	8x8	2.2.0
	8x22B		32K	MCore	8x8
			64K	MCore	8x8
DeepSeek-V2	236B	deepseek2	8K	MCore	20x8	2.2.0
DeepSeek-V2-coder	236B	deepseek2_coder	8K	MCore	20x8	2.2.0
DeepSeek-V2-Lite	16B	deepseek2_lite	8K	MCore	1x8	2.3.0
DeepSeek-V2.5	236B	deepseek25	8K	MCore	20x8	In progress
DeepSeek-V3	671B	deepseek3	4K	MCore	64x8
MiniCPM	8x2B	minicpm	4K	MCore	1x8	2.2.0
Phi3.5	MoE-instruct	phi35	4K	MCore	2x8
GLM4.5	106B	glm45-moe	4K	MCore	8x16