报错原因是 Megatron 的 helpers C++ 扩展没有编译。进到对应目录手动 make 一下就行：

cd Megatron-LM/megatron/core/datasets/
make

如果 make 报缺 pybind11，先 pip install pybind11 再编译。编译完成后该目录下会生成 helpers_cpp*.so，再跑预训练就不会报这个错了。
另外提醒一下：你当前使用的 MindSpeed-LLM 2.1.0 + MindSpeed-Core-MS r0.3.0 已经过了维护周期，后续不再有 bug 修复和新特性更新。建议切换到最新的维护分支，可以参考版本维护策略，版本配套关系参考最新分支下的 install_guide.md。

报错原因是 Megatron 的 helpers C++ 扩展没有编译。进到对应目录手动 make 一下就行：

cd Megatron-LM/megatron/core/datasets/
make

如果 make 报缺 pybind11，先 pip install pybind11 再编译。编译完成后该目录下会生成 helpers_cpp*.so，再跑预训练就不会报这个错了。
另外提醒一下：你当前使用的 MindSpeed-LLM 2.1.0 + MindSpeed-Core-MS r0.3.0 已经过了维护周期，后续不再有 bug 修复和新特性更新。建议切换到最新的维护分支，可以参考版本维护策略，版本配套关系参考最新分支下的 install_guide.md。

@HANHU1CHEN
参考[版本维护策略]重新安装 MindSpeed-LLM + MindSpeed-Core-MS ，执行qwen25-7b预训练，报错：

报错原因是 Megatron 的 helpers C++ 扩展没有编译。进到对应目录手动 make 一下就行：

cd Megatron-LM/megatron/core/datasets/
make

如果 make 报缺 pybind11，先 pip install pybind11 再编译。编译完成后该目录下会生成 helpers_cpp*.so，再跑预训练就不会报这个错了。
另外提醒一下：你当前使用的 MindSpeed-LLM 2.1.0 + MindSpeed-Core-MS r0.3.0 已经过了维护周期，后续不再有 bug 修复和新特性更新。建议切换到最新的维护分支，可以参考版本维护策略，版本配套关系参考最新分支下的 install_guide.md。

@HANHU1CHEN
参考[版本维护策略]重新安装 MindSpeed-LLM + MindSpeed-Core-MS ，执行qwen25-7b预训练，报错：The result of multiplication calculation is correct, MindSpore has been installed on platform [Ascend] successfully!
(qwen2.5-lora) root@k8s-node1:/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM# tail -f worker_0.log
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/core/transformer/multi_token_prediction.py", line 29, in
from mindspeed_llm.tasks.models.transformer.deepseek4.mhc.mhc import get_mhc_spec, hc_repeat
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/models/transformer/deepseek4/mhc/mhc.py", line 43, in
from mindspeed_llm.tasks.models.transformer.deepseek4.rmsnorm_without_weight import rmsnorm_without_weight_triton
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/models/transformer/deepseek4/rmsnorm_without_weight.py", line 6, in
from mindspeed.lite.ops.triton.utils import autocast_custom_bwd, autocast_custom_fwd, input_guard
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed/mindspeed/lite/ops/triton/utils.py", line 176, in
assert device == 'cuda', 'Only cuda device is supported for PyTorch version < 2.4.0.'
^^^^^^^^^^^^^^^^
AssertionError: Only cuda device is supported for PyTorch version < 2.4.0.

  如果不安装Triton，会报错：ModuleNotFoundError: No module named 'triton'

请问你是要使用mindspore还是torch后端进行训练呢？我看你的报错信息有MindSpore相关的提示，请参考对应的安装文档：pytorch训练安装文档或者mindspore训练安装文档.

如果是缺少triton，请安装最新版的triton后重试。

请问你是要使用mindspore还是torch后端进行训练呢？我看你的报错信息有MindSpore相关的提示，请参考对应的安装文档：pytorch训练安装文档或者mindspore训练安装文档.

如果是缺少triton，请安装最新版的triton后重试。

@HANHU1CHEN

使用mindspore后端进行训练，环境就是参考MindSpore进行安装
参考链接：https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/mindspore/install_guide.md
最新版的triton安装链接有吗？还有这个问题和cann和驱动版本有强关联性吗

请参考triton-ascend仓库进行安装

请参考triton-ascend仓库进行安装

@HANHU1CHEN

使用pip install 方式安装了triton-ascend，执行预训练仍然报错：

请参考triton-ascend仓库进行安装

@HANHU1CHEN

使用pip install 方式安装了triton-ascend，执行预训练仍然报错：

Traceback (most recent call last):
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/pretrain_gpt.py", line 10, in
from mindspeed_llm import megatron_adaptor
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/init.py", line 19, in
from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 114, in
FeatureAdaptor.execute()
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 89, in execute
MindSpeedFeaturesManager.apply_features_patches(args)
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed/mindspeed/features_manager/features_manager.py", line 28, in apply_features_patches
feature.register_patches(MindSpeedPatchesManager, mindspeed_args)
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/features_manager/transformer/mtp.py", line 20, in register_patches
from mindspeed_llm.core.transformer.multi_token_prediction import get_mtp_num_layers_to_build,
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/core/transformer/multi_token_prediction.py", line 29, in
from mindspeed_llm.tasks.models.transformer.deepseek4.mhc.mhc import get_mhc_spec, hc_repeat
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/models/transformer/deepseek4/mhc/mhc.py", line 43, in
from mindspeed_llm.tasks.models.transformer.deepseek4.rmsnorm_without_weight import rmsnorm_without_weight_triton
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/tasks/models/transformer/deepseek4/rmsnorm_without_weight.py", line 6, in
from mindspeed.lite.ops.triton.utils import autocast_custom_bwd, autocast_custom_fwd, input_guard
File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed/mindspeed/lite/ops/triton/utils.py", line 176, in
assert device == 'cuda', 'Only cuda device is supported for PyTorch version < 2.4.0.'
^^^^^^^^^^^^^^^^
AssertionError: Only cuda device is supported for PyTorch version < 2.4.0.

注释掉MindSpeed-Core-MS/MindSpeed/mindspeed/lite/ops/triton/utils.py中

assert device == 'cuda', 'Only cuda device is supported for PyTorch version < 2.4.0.'

def check_pytorch_version(version_s: str = '2.4') -> bool:
return version.parse(torch.version) >= version.parse(version_s)

if check_pytorch_version('2.4'):
device = 'cuda' if device == 'cpu' else device
autocast_custom_fwd = functools.partial(torch.amp.custom_fwd, device_type=device)
autocast_custom_bwd = functools.partial(torch.amp.custom_bwd, device_type=device)

def custom_device_ctx(index: int):
    return device_torch_lib.device(index)

else:

assert device == 'cuda', 'Only cuda device is supported for PyTorch version < 2.4.0.'

autocast_custom_fwd = device_torch_lib.amp.custom_fwd
autocast_custom_bwd = device_torch_lib.amp.custom_bwd


def custom_device_ctx(index: int):
    return torch.cuda.device(index)

执行报错：
[rank0]: Traceback (most recent call last):
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/pretrain_gpt.py", line 309, in
[rank0]: main()
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/training/utils.py", line 1136, in wrapper
[rank0]: return func(*args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/pretrain_gpt.py", line 302, in main
[rank0]: pretrain(train_valid_test_datasets_provider,
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/training/training.py", line 502, in pretrain
[rank0]: train_args, test_data_iterator_list = build_train_args(args, timers, train_valid_test_dataset_provider,
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/training/training.py", line 360, in build_train_args
[rank0]: model, optimizer, opt_param_scheduler = setup_model_and_optimizer(
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/Megatron-LM/megatron/training/training.py", line 1165, in setup_model_and_optimizer
[rank0]: args.iteration, args.num_floating_point_operations_so_far = load_checkpoint(
[rank0]: ^^^^^^^^^^^^^^^^
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MindSpeed-LLM/mindspeed_llm/training/checkpointing.py", line 126, in wrapper
[rank0]: return fn(ddp_model, optimizer, opt_param_scheduler, strict=strict, *args, **kwargs)
[rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/Megatron-LM/megatron/training/checkpointing.py", line 1378, in load_checkpoint
[rank0]: ddp_model[0].load_state_dict(state_dict['model'], strict=strict)
[rank0]: File "/home/openlab/zll/MindSpeed-Core-MS/MSAdapter/msadapter/nn/modules/module.py", line 1230, in load_state_dict
[rank0]: raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
[rank0]: RuntimeError: Error(s) in loading state_dict for GPTModel:
[rank0]: Missing key(s) in state_dict: "decoder.layers.0.self_attention.linear_qkv.bias", "decoder.layers.0.mlp.linear_fc1.layer_norm_weight", "decoder.layers.1.self_attention.linear_qkv.bias", "decoder.layers.1.mlp.linear_fc1.layer_norm_weight", "decoder.layers.2.self_attention.linear_qkv.bias", "decoder.layers.2.mlp.linear_fc1.layer_norm_weight", "decoder.layers.3.self_attention.linear_qkv.bias", "decoder.layers.3.mlp.linear_fc1.layer_norm_weight", "decoder.layers.4.self_attention.linear_qkv.bias", "decoder.layers.4.mlp.linear_fc1.layer_norm_weight", "decoder.layers.5.self_attention.linear_qkv.bias", "decoder.layers.5.mlp.linear_fc1.layer_norm_weight", "decoder.layers.6.self_attention.linear_qkv.bias", "decoder.layers.6.mlp.linear_fc1.layer_norm_weight", "decoder.layers.7.self_attention.linear_qkv.bias", "decoder.layers.7.mlp.linear_fc1.layer_norm_weight", "decoder.layers.8.self_attention.linear_qkv.bias", "decoder.layers.8.mlp.linear_fc1.layer_norm_weight", "decoder.layers.9.self_attention.linear_qkv.bias", "decoder.layers.9.mlp.linear_fc1.layer_norm_weight", "decoder.layers.10.self_attention.linear_qkv.bias", "decoder.layers.10.mlp.linear_fc1.layer_norm_weight", "decoder.layers.11.self_attention.linear_qkv.bias", "decoder.layers.11.mlp.linear_fc1.layer_norm_weight", "decoder.layers.12.self_attention.linear_qkv.bias", "decoder.layers.12.mlp.linear_fc1.layer_norm_weight", "decoder.layers.13.self_attention.linear_qkv.bias", "decoder.layers.13.mlp.linear_fc1.layer_norm_weight".
[rank0]: Unexpected key(s) in state_dict: "decoder.layers.0.pre_mlp_layernorm.weight", "decoder.layers.1.pre_mlp_layernorm.weight", "decoder.layers.2.pre_mlp_layernorm.weight", "decoder.layers.3.pre_mlp_layernorm.weight", "decoder.layers.4.pre_mlp_layernorm.weight", "decoder.layers.5.pre_mlp_layernorm.weight", "decoder.layers.6.pre_mlp_layernorm.weight", "decoder.layers.7.pre_mlp_layernorm.weight", "decoder.layers.8.pre_mlp_layernorm.weight", "decoder.layers.9.pre_mlp_layernorm.weight", "decoder.layers.10.pre_mlp_layernorm.weight", "decoder.layers.11.pre_mlp_layernorm.weight", "decoder.layers.12.pre_mlp_layernorm.weight", "decoder.layers.13.pre_mlp_layernorm.weight".

请参考triton-ascend仓库进行安装

@HANHU1CHEN

启动脚本中修改增加了如下参数： --model-type-hf llama2 ，执行报错：
[rank0]: RuntimeError: Error(s) in loading state_dict for GPTModel:
[rank0]: Missing key(s) in state_dict: "decoder.layers.0.self_attention.linear_qkv.bias", "decoder.layers.0.mlp.linear_fc1.layer_norm_weight", "decoder.layers.1.self_attention.linear_qkv.bias", "decoder.layers.1.mlp.linear_fc1.layer_norm_weight", "decoder.layers.2.self_attention.linear_qkv.bias", "decoder.layers.2.mlp.linear_fc1.layer_norm_weight", "decoder.layers.3.self_attention.linear_qkv.bias", "decoder.layers.3.mlp.linear_fc1.layer_norm_weight", "decoder.layers.4.self_attention.linear_qkv.bias", "decoder.layers.4.mlp.linear_fc1.layer_norm_weight", "decoder.layers.5.self_attention.linear_qkv.bias", "decoder.layers.5.mlp.linear_fc1.layer_norm_weight", "decoder.layers.6.self_attention.linear_qkv.bias", "decoder.layers.6.mlp.linear_fc1.layer_norm_weight", "decoder.layers.7.self_attention.linear_qkv.bias", "decoder.layers.7.mlp.linear_fc1.layer_norm_weight", "decoder.layers.8.self_attention.linear_qkv.bias", "decoder.layers.8.mlp.linear_fc1.layer_norm_weight", "decoder.layers.9.self_attention.linear_qkv.bias", "decoder.layers.9.mlp.linear_fc1.layer_norm_weight", "decoder.layers.10.self_attention.linear_qkv.bias", "decoder.layers.10.mlp.linear_fc1.layer_norm_weight", "decoder.layers.11.self_attention.linear_qkv.bias", "decoder.layers.11.mlp.linear_fc1.layer_norm_weight", "decoder.layers.12.self_attention.linear_qkv.bias", "decoder.layers.12.mlp.linear_fc1.layer_norm_weight", "decoder.layers.13.self_attention.linear_qkv.bias", "decoder.layers.13.mlp.linear_fc1.layer_norm_weight".
[rank0]: Unexpected key(s) in state_dict: "decoder.layers.0.pre_mlp_layernorm.weight", "decoder.layers.1.pre_mlp_layernorm.weight", "decoder.layers.2.pre_mlp_layernorm.weight", "decoder.layers.3.pre_mlp_layernorm.weight", "decoder.layers.4.pre_mlp_layernorm.weight", "decoder.layers.5.pre_mlp_layernorm.weight", "decoder.layers.6.pre_mlp_layernorm.weight", "decoder.layers.7.pre_mlp_layernorm.weight", "decoder.layers.8.pre_mlp_layernorm.weight", "decoder.layers.9.pre_mlp_layernorm.weight", "decoder.layers.10.pre_mlp_layernorm.weight", "decoder.layers.11.pre_mlp_layernorm.weight", "decoder.layers.12.pre_mlp_layernorm.weight", "decoder.layers.13.pre_mlp_layernorm.weight".

不添加这个 参数--model-type-hf llama2，执行报这个错：
RuntimeError: Cannot infer model type from architectures of Huggingface config.json '{arch_row}'. Please specify --model-type-hf explicitly.
当前环境中只支持这些类型--model-type-hf
ValueError: Current --enable-hf2mg-convert does not support model type 'qwen'. Supported models: llama2, qwen3, qwen3-moe, deepseek3, glm45-air, glm45, bailing_mini, qwen3-next, seed-oss, deepseek32, magistral, deepseek2-lite, mamba2

模型的config.json文件如下：
{
"architectures": [
"Qwen2ForCausalLM"
],
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151643,
"hidden_act": "silu",
"hidden_size": 3584,
"initializer_range": 0.02,
"intermediate_size": 18944,
"max_position_embeddings": 131072,
"max_window_layers": 28,
"model_type": "qwen2",
"num_attention_heads": 28,
"num_hidden_layers": 28,
"num_key_value_heads": 4,
"rms_norm_eps": 1e-06,
"rope_theta": 1000000.0,
"sliding_window": 131072,
"tie_word_embeddings": false,
"torch_dtype": "float16",
"transformers_version": "4.40.1",
"use_cache": true,
"use_mrope": false,
"use_sliding_window": false,
"vocab_size": 152064
}

请把你运行的具体脚本发一下，我们这边方便复现结果

请把你运行的具体脚本发一下，我们这边方便复现结果

@HANHU1CHEN

执行的训练脚本：
#!/bin/bash
#export TRITON_SKIP_CUDA_CHECK=1
export CUDA_DEVICE_MAX_CONNECTIONS=1

NPUS_PER_NODE=4
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=4
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

please fill these path configurations

CKPT_LOAD_DIR="./model_from_hf/qwen2.5_7b_hf/"
CKPT_SAVE_DIR="./model_weights/qwen2.5_mcore/"
DATA_PATH="./dataset/alpaca_text_document"
TOKENIZER_PATH="./model_from_hf/qwen2.5_7b_hf/"

TP=2
PP=2
SEQ_LEN=1024
MBS=1
GBS=4

DISTRIBUTED_ARGS="
--worker_num $WORLD_SIZE
--local_worker_num $NPUS_PER_NODE
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT
--node_rank $NODE_RANK
--join=False
"

GPT_ARGS="
--use-mcore-models
--tensor-model-parallel-size ${TP}
--pipeline-model-parallel-size ${PP}
--sequence-parallel
--num-layers 28
--hidden-size 3584
--ffn-hidden-size 18944
--num-attention-heads 28
--max-position-embeddings ${SEQ_LEN}
--seq-length ${SEQ_LEN}
--disable-bias-linear
--add-qkv-bias
--group-query-attention
--num-query-groups 4
--use-flash-attn
--swiglu
--use-fused-swiglu
--normalization RMSNorm
--norm-epsilon 1e-6
--use-fused-rmsnorm
--position-embedding-type rope
--rotary-base 1000000
--use-fused-rotary-pos-emb
--untie-embeddings-and-output-weights
--micro-batch-size ${MBS}
--global-batch-size ${GBS}
--make-vocab-size-divisible-by 1
--padded-vocab-size 152064
--tokenizer-type PretrainedFromHF
--tokenizer-name-or-path ${TOKENIZER_PATH}
--attention-dropout 0.0
--hidden-dropout 0.0
--train-iters 100
--lr 1.25e-6
--lr-decay-style cosine
--min-lr 1.25e-7
--lr-warmup-fraction 0.01
--init-method-std 0.01
--weight-decay 1e-1
--clip-grad 1.0
--adam-beta1 0.9
--adam-beta2 0.95
--initial-loss-scale 4096
--no-gradient-accumulation-fusion
--no-masked-softmax-fusion
--attention-softmax-in-fp32
--bf16
--ckpt-format torch
"

DATA_ARGS="
--data-path $DATA_PATH
--split 100,0,0
"

CKPT_ARGS="
--no-load-optim
--no-load-rng
--no-save-optim
--no-save-rng
--seed 1234
"

OUTPUT_ARGS="
--log-interval 1
--save-interval 2000
--eval-interval 2000
--eval-iters 0
--log-throughput
"

msrun $DISTRIBUTED_ARGS pretrain_gpt.py
$GPT_ARGS
$DATA_ARGS
$CKPT_ARGS
$OUTPUT_ARGS
--model-type-hf llama2
--distributed-backend nccl
--load ${CKPT_LOAD_DIR}
--save ${CKPT_SAVE_DIR}
--ai-framework mindspore
| tee logs/pretrain_qwen25_7b_32k.log

感谢您的反馈和建议 。这里需要提前同步一个情况：MindSpore 后端相关的文档与框架代码计划后续下架，不再继续维护，因此这部分相关内容我们不会再做更新了，给您带来的不便还请谅解。
如果您有 LLM 训练相关的需求，推荐关注/迁移到仓库当前主推的pytorch后端（Megatron / FSDP2 方向），相关文档会持续维护。再次感谢您的关注！