在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

⚠️ 安全信息提醒：请仔细检查提供的文本内容，确保其不包含敏感数据信息，包括但不限于：

• API 令牌或密钥
• 密码或身份验证凭证
• 私有网址或接口地址
• 个人或机密数据
• ...

在分享配置信息或代码示例时，请将敏感信息脱敏处理，或使用 <TOKEN> 等占位符替代原有内容。

环境信息

使用官方镜像：swr.cn-south-1.myhuaweicloud.com/ascendhub/mindspeed-llm:openeuler22.03-mindspeed-llm-2.3.0-a2-arm
- 操作系统：openEuler 22.03 (LTS-SP3)
- 昇腾硬件信息：910B3
- CANN软件版本：8.5.0
- 安装的对应软件版本：torch ==2.7.1，torch_npu==2.7.1.post2，python3.10.19

🐛 问题描述

master分支使用fsdp2拉起qwen3-14b，报错RuntimeError: the patch of compile exist !

Traceback (most recent call last):
  File "/home/xxx/dev_exp/FSDP/MindSpeed-LLM/train_fsdp2.py", line 27, in <module>
    from mindspeed_llm.fsdp2.models.model_factory import ModelFactory
  File "/home/xxx/dev_exp/FSDP/MindSpeed-LLM/mindspeed_llm/__init__.py", line 19, in <module>
    from mindspeed_llm.tasks import megatron_adaptor_v2 as megatron_adaptor
  File "/home/xxx/dev_exp/FSDP/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 92, in <module>
    FeatureAdaptor.execute()
  File "/home/xxx/dev_exp/FSDP/MindSpeed-LLM/mindspeed_llm/tasks/megatron_adaptor_v2.py", line 84, in execute
    MindSpeedFeaturesManager.apply_features_pre_patches(args)
  File "/home/xxx/dev_exp/FSDP/MindSpeed/mindspeed/features_manager/features_manager.py", line 20, in apply_features_pre_patches
    feature.pre_register_patches(MindSpeedPatchesManager, mindspeed_args)
  File "/home/xxx/dev_exp/FSDP/MindSpeed-LLM/mindspeed_llm/features_manager/megatron_basic/requirements_basic.py", line 24, in pre_register_patches
    super().pre_register_patches(patch_manager, args)
  File "/home/xxx/dev_exp/FSDP/MindSpeed/mindspeed/features_manager/megatron_basic/requirements_basic.py", line 24, in pre_register_patches
    self.te_adaptation(patch_manager, args)
  File "/home/xxx/dev_exp/FSDP/MindSpeed/mindspeed/features_manager/megatron_basic/requirements_basic.py", line 32, in te_adaptation
    pm.register_patch('torch.compile', dummy_compile)
  File "/home/xxx/dev_exp/FSDP/MindSpeed/mindspeed/patch_utils.py", line 186, in register_patch
    MindSpeedPatchesManager.patches_info.get(orig_func_name).set_patch_func(new_func, force_patch)
  File "/home/xxx/dev_exp/FSDP/MindSpeed/mindspeed/patch_utils.py", line 80, in set_patch_func
    raise RuntimeError('the patch of {} exist !'.format(self.orig_func_name))
RuntimeError: the patch of compile exist !

训练脚本

cat examples/fsdp2/qwen3/pretrain_qwen3_14b_4k_fsdp2.sh
#!/bin/bash

export CUDA_DEVICE_MAX_CONNECTIONS=2
export HCCL_CONNECT_TIMEOUT=1800
export TASK_QUEUE_ENABLE=2
export CPU_AFFINITY_CONF=1
export TORCH_HCCL_ZERO_COPY=1
export MULTI_STREAM_MEMORY_REUSE=2
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

NPUS_PER_NODE=8
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=0
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

# please fill these path configurations
DATA_PATH="./datasets/train-00000-of-00001-a09b74b3ef9c3b56.parquet"
TOKENIZER_PATH="/home/model_weights/Qwen3-14B"
HF_PATH="/home/model_weights/Qwen3-14B"
SAVE_CKPT_DIR="./ckpt"
FSDP2_PATH="./examples/fsdp2/qwen3/fsdp2_config.yaml"


TP=1
PP=1
EP=1
CP=1
MBS=4
GBS=128

SEQ_LENGTH=4096
TRAIN_ITERS=2000

time=$(date "+%Y%m%d-%H%M%S")

DISTRIBUTED_ARGS="
    --nproc_per_node $NPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

FSDP2_ARGS="
    --use-torch-fsdp2 \
    --untie-embeddings-and-output-weights \
    --ckpt-format torch_dcp \
    --fsdp2-config-path $FSDP2_PATH \
    --init-from-hf-path $HF_PATH \
    --model-id qwen3 \
    --loss-compute-mode chunk \
    --loss-chunk-size 1024 \
"

OPTIMIZE_ARGS="
    --use-fused-rotary-pos-emb \
    --use-fused-rmsnorm \
    --no-masked-softmax-fusion \
    --no-gradient-accumulation-fusion \
"

TRAIN_ARGS="
    --micro-batch-size ${MBS} \
    --global-batch-size ${GBS} \
    --lr 1.25e-5 \
    --lr-decay-style cosine \
    --min-lr 1.25e-7 \
    --weight-decay 1e-1 \
    --lr-warmup-fraction 0.01 \
    --attention-dropout 0.0 \
    --init-method-std 0.01 \
    --hidden-dropout 0.0 \
    --clip-grad 1.0 \
    --adam-beta1 0.9 \
    --adam-beta2 0.95 \
    --initial-loss-scale 4096 \
    --seed 42 \
    --bf16 \
    --train-iters ${TRAIN_ITERS} \
    --seq-length ${SEQ_LENGTH} \
    --no-shared-storage \
    --tokenizer-not-use-fast \
"

MODEL_ARGS="
    --num-layers 40 \
    --hidden-size 5120 \
    --num-attention-heads 40 \
    --group-query-attention \
    --num-query-groups 8 \
    --max-position-embeddings ${SEQ_LENGTH} \
    --tokenizer-type PretrainedFromHF \
    --tokenizer-name-or-path ${TOKENIZER_PATH} \
"

DATA_ARGS="
    --data-path $DATA_PATH \
    --split 100,0,0 \
    --handler-name GeneralPretrainHandler \
"

OUTPUT_ARGS="
    --log-interval 1 \
    --save-interval ${TRAIN_ITERS} \
    --eval-interval ${TRAIN_ITERS} \
    --eval-iters 0 \
    --no-load-optim \
    --no-load-rng \
    --save ${SAVE_CKPT_DIR}
"

torchrun $DISTRIBUTED_ARGS train_fsdp2.py \
    $DATA_ARGS \
    $OUTPUT_ARGS \
    $MODEL_ARGS \
    $OPTIMIZE_ARGS \
    $TRAIN_ARGS \
    $FSDP2_ARGS \
    --distributed-backend nccl \
    | tee logs/train_fsdp2_qwen3_14b_seqlen${SEQ_LENGTH}_A2-${time}.log

欢迎加入社区，感谢您对社区的贡献 🎉!