[Usage]: sft_trainer.py中会不会有除0报错的可能-MindSpeed-LLM-AtomGit

已关闭

[Usage]: sft_trainer.py中会不会有除0报错的可能 #1329

KL4805创建于 5月12日关闭于 6月1日

5月12日创建

在提交新问题之前，请确保您已经在社区中搜索过相关问题，并使用了社区中提供的资源/工具后，仍未找到满意的解决方式。

环境信息

例如：
- 操作系统 Ubuntu 22.04
- 昇腾硬件信息
- CANN软件版本 8.5.0
- 安装的对应软件版本 2.3.0

使用场景及问题

我在看sft_trainer.py的时候发现了这样一段代码

https://gitcode.com/Ascend/MindSpeed-LLM/blob/2.3.0/mindspeed_llm/tasks/posttrain/sft/sft_trainer.py#L157

会不会存在这样一种情况，就是一个卡分到的序列，全是masked_tokens，比如一条特别长的input。在这种情况下，loss_mask_sum = 0，会不会引发除0报错？

举个例子，序列长度65536，开启CP=4，每卡分到16384 tokens，假设有一条数据，指令部分20000 tokens，那么就可能导致一个卡上的loss_mask_sum = 0，会导致报错吗?

欢迎加入社区，感谢您对社区的贡献 🎉!

ascend-robot成员

5月12日添加了label：usage

K KL4805

5月12日修改标题为 “[Usage]: sft_trainer.py中会不会有除0报错的可能”，原标题为“[Usage]: ”

K KL4805

5月12日修改了issue 的描述

HANHU1CHEN成员

5月12日评论：

你好，这段代码不会触发除零。

loss = torch.cat([torch.sum(losses * loss_mask).view(1), loss_mask.sum().view(1)])
torch.distributed.all_reduce(loss, group=mpu.get_context_parallel_group())
loss_sum = loss[0]
loss_mask_sum = loss[1]
...
loss = loss_sum / loss_mask_sum

loss_mask_sum 在除法时已经是 CP 组聚合后的值，即整条序列上所有未 mask token 的总数。你那个例子里 instruction 20000 / response 45536，单卡 local loss_mask.sum() 即使为 0，all-reduce 之后 loss_mask_sum = 45536，除法照常。

H HANHU1CHEN成员

5月12日将 HANHU1CHEN 设为负责人

H HANHU1CHEN成员

5月12日添加了label：resolved

KL4805

5月12日评论：

那么如果我考虑以下这种情况：

开启sequence-parallel，序列维度进一步被切分。假设我TP=4, CP=2，开启sequence-parallel

CP组1：NPU0, NPU1, NPU2, NPU3
CP组2：NPU4, NPU5, NPU6, NPU7

聚合方式应该是NPU0与NPU4聚合，NPU1与NPU5聚合，NPU2与NPU6聚合，NPU3与NPU7聚合。那么如果有一条数据seq-length=65536，最后50000个token全部都是mask（比如说因为packing），那么会不会在NPU3与NPU7（均分到全mask数据）聚合时候报除以0的错误？

我这么问，是因为我们在SFT训练时遇到了NaN的问题，但是只要我在数据中手动drop掉最后一条数据（一般都是pack的剩余，有大量的尾部mask），训练就可以正常进行，让我怀疑这个问题。

HANHU1CHEN成员

5月12日评论：

感谢反馈。我们需要完整的脚本来复现这个问题，请提供：

完整运行脚本

启动脚本（含所有并行配置：TP / CP / PP / DP、--sequence-parallel、--reset-attention-mask、--reset-position-ids 等）

环境信息

MindSpeed-LLM commit / 分支
MindSpeed-Core、Megatron 版本
CANN、torch_npu版本

数据与复现

数据集来源、预处理脚本、--prompt-type / template 配置
触发 NaN 的具体 step 或样本特征（你提到 drop 最后一条就正常，那条数据的 input_ids / loss_mask 长度分布能否提供）
完整的 NaN 报错日志（含 stack trace 和 check_for_nan_in_loss_and_grad 的输出）

除此之外，是否对比过关闭 CP（CP=1）、关闭 sequence-parallel 时是否仍然 NaN。

H HANHU1CHEN成员

5月12日将 yanzhixiao23 设为负责人

H HANHU1CHEN成员

5月12日移除了负责人 HANHU1CHEN

H HANHU1CHEN成员

5月12日添加了label：pending；删除了label：resolved

KL4805

5月12日评论：

完整运行脚本（被训练模型Qwen3-Coder-30B-A3B-Instruct）

#!/bin/bash
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NPU_ASD_ENABLE=0
export WITHOUT_JIT_COMPILE=1
export HCCL_CONNECT_TIMEOUT=7200
export MOX_FILE_LARGE_FILE_TASK_NUM=2

MASTER_HOST="$VC_WORKER_HOSTS"
MASTER_ADDR="${VC_WORKER_HOSTS%%,*}"
MASTER_PORT=6000
JOB_ID="1234"
NNODES="$MA_NUM_HOSTS"
NODE_RANK="$VC_TASK_INDEX"
NPUS_PER_NODE="$MA_NUM_GPUS"
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

if [[ $NODE_RANK == 0 ]]; then
    EXT_ARGS="--rdzv_conf=is_host=1"
else
    EXT_ARGS=""
fi

TP=4
PP=4
EP=8
CP=2
CP_TYPE='ulysses_cp_algo'

SEQ_LENGTH=65536
TRAIN_ITERS=2500
SAVE_ITERS=300
MBS=1
GBS=16

SCRIPT_NAME="tune_qwen3_30b_a3b_base_tulu3"

CKPT_LOAD_DIR="/mnt/efs/mg_checkpoint_dir/Qwen3-Coder-30B-A3B-Instruct-mcore-TP${TP}PP${PP}EP${EP}"
TOKENIZER_PATH="/mnt/efs/tokenizer_dir/Qwen3-Coder-30B-A3B-Instruct/"
CKPT_SAVE_DIR="/home/ma-user/modelarts/outputs/train_output_0/"
TENSORBOARD_PATH="/home/ma-user/modelarts/outputs/train_output_0/tensorboard_dir/"
LOG_PATH="/mnt/efs/jyl/logs/${SCRIPT_NAME}.log"

# total 9110
# GBS 32, 285 iters / epoch
# GBS 16, 570 iters / epoch
DATA_PATH="179 /mnt/efs/jyl/data_dir/sft_data/64k_pack/dataset_open_tulu3/dataset_open_tulu3"
# DATA_PATH+=" 5129 /mnt/efs/jyl/data_dir/sft_data/64k_pack/dataset_open_lmsys_gpt_oss_sft_en_droplast/dataset_open_lmsys_gpt_oss_sft_en_droplast"


DISTRIBUTED_ARGS="
    --nproc_per_node $NPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

MOE_ARGS="
    --num-experts 128 \
    --expert-tensor-parallel-size 1 \
    --moe-router-topk 8 \
    --moe-ffn-hidden-size 768 \
    --moe-grouped-gemm \
    --moe-permutation-async-comm \
    --moe-permute-fusion \
    --moe-alltoall-overlap-comm \
    --moe-token-dispatcher-type alltoall \
    --moe-router-load-balancing-type aux_loss \
    --moe-layer-freq -1 \
    --first-k-dense-replace -1 \
    --moe-aux-loss-coeff 0.001
"

OPTIMIZE_ARGS="
    --use-flash-attn \
    --use-fused-rotary-pos-emb \
    --sequence-parallel \
    --use-rotary-position-embeddings \
    --use-fused-swiglu \
    --use-fused-rmsnorm \
    --no-masked-softmax-fusion \
    --use-distributed-optimizer \
    --swap-optimizer \
    --swap-attention \
    --gemm-gradient-accumulation-fusion \
    --recompute-granularity full \
    --recompute-method uniform \
    --recompute-num-layers 1
"

TRAIN_ARGS="
    --reset-attention-mask \
    --attention-mask-type general \
    --micro-batch-size ${MBS} \
    --global-batch-size ${GBS} \
    --lr 5e-7 \
    --lr-decay-style cosine \
    --min-lr 5e-8 \
    --weight-decay 1e-1 \
    --lr-warmup-fraction 0.03 \
    --attention-dropout 0.0 \
    --init-method-std 0.01 \
    --hidden-dropout 0.0 \
    --clip-grad 1.0 \
    --adam-beta1 0.9 \
    --adam-beta2 0.95 \
    --initial-loss-scale 4096 \
    --seed 42 \
    --bf16 \
    --train-iters ${TRAIN_ITERS} \
    --seq-length ${SEQ_LENGTH} \
    --manual-gc \
    --manual-gc-interval 50
"

MODEL_PARALLEL_ARGS="
    --tensor-model-parallel-size ${TP} \
    --pipeline-model-parallel-size ${PP} \
    --expert-model-parallel-size ${EP} \
    --context-parallel-size ${CP} \
    --context-parallel-algo ${CP_TYPE}
"

GPT_ARGS="
    --use-mcore-models \
    --spec mindspeed_llm.tasks.models.spec.qwen3_spec layer_spec \
    --kv-channels 128 \
    --qk-layernorm \
    --norm-topk-prob \
    --tokenizer-name-or-path ${TOKENIZER_PATH} \
    --max-position-embeddings ${SEQ_LENGTH} \
    --num-layers 48 \
    --hidden-size 2048 \
    --ffn-hidden-size 6144 \
    --num-attention-heads 32 \
    --tokenizer-type PretrainedFromHF \
    --make-vocab-size-divisible-by 1 \
    --padded-vocab-size 151936 \
    --rotary-base 1000000 \
    --untie-embeddings-and-output-weights \
    --disable-bias-linear \
    --position-embedding-type rope \
    --normalization RMSNorm \
    --swiglu \
    --attention-softmax-in-fp32 \
    --no-gradient-accumulation-fusion \
    --group-query-attention \
    --num-query-groups 4 \
    --tensorboard-dir ${TENSORBOARD_PATH} \
    --tensorboard-log-interval 10
"

DATA_ARGS="
    --data-path $DATA_PATH \
    --split 100,0,0
"

OUTPUT_ARGS="
    --log-interval 1 \
    --save-interval ${SAVE_ITERS} \
    --eval-interval ${SAVE_ITERS} \
    --eval-iters 0 \
    --no-load-optim \
    --load ${CKPT_LOAD_DIR} \
    --save ${CKPT_SAVE_DIR} \
    --no-load-rng \
    --log-throughput
"

TUNE_ARGS="
    --finetune \
    --stage sft \
    --is-instruction-dataset \
    --tokenizer-not-use-fast \
    --prompt-type qwen3 \
    --neat-pack \
    --pack \
    --reset-position-ids
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
    $TUNE_ARGS \
    $GPT_ARGS \
    $DATA_ARGS \
    $MOE_ARGS \
    $OUTPUT_ARGS \
    $OPTIMIZE_ARGS \
    $TRAIN_ARGS \
    $MODEL_PARALLEL_ARGS \
    --distributed-backend nccl \
    | tee ${LOG_PATH}

环境信息
MindSpeed-LLM 2.3.0, MindSpeed0.12.1，Megatron参考的2.3.0的安装要求切换的分支。
torch_npu 2.7.1post2，CANN版本 8.3.RC1
数据
数据来自于tulu3-instruction-following: https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following
预处理使用的选项是：alpaca handler，neat-pack, pack均打开，上下文长度65536. prompt-type Qwen3

我无法给出具体的保存信息，因为modelarts平台没有给具体的报错信息，只有这个：


W0510 09:24:21.680845 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1171 closing signal SIGTERM
W0510 09:24:21.693475 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1172 closing signal SIGTERM
W0510 09:24:21.696840 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1173 closing signal SIGTERM
W0510 09:24:21.699285 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1174 closing signal SIGTERM
W0510 09:24:21.701289 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1175 closing signal SIGTERM
W0510 09:24:21.719487 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1177 closing signal SIGTERM
W0510 09:24:21.720776 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1178 closing signal SIGTERM
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
E0510 09:24:30.336779 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -11) local_rank: 5 (pid: 1176) of binary: /home/ma-user/miniconda3/envs/mindspeed-230-new/bin/python3.10
Traceback (most recent call last):
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/bin/torchrun", line 6, in <module>
    sys.exit(main())
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
    return f(*args, **kwargs)
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in main
    run(args)
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/run.py", line 883, in run
    elastic_launch(
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
    return launch_agent(self._config, self._entrypoint, list(args))
  File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
    raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
posttrain_gpt.py FAILED
------------------------------------------------------------
Failures:
  <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
  time      : 2026-05-10_09:24:21
  host      : ma-job-555a0749-cfed-45b6-84b1-29d7897832c5-worker-0.ma-job-555a0749-cfed-45b6-84b1-29d7897832c5.os-d910b-dev-codemate.svc.cluster.local
  rank      : 5 (local_rank: 5)
  exitcode  : -11 (pid: 1176)
  error_file: <N/A>
  traceback : Signal 11 (SIGSEGV) received by PID 1176
============================================================
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
[ERROR] 2026-05-10-09:24:30 (PID:949, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

训练配置是64卡910B3.

关于这条数据的特征，因为有点长，我无法给出具体的分布，但是有这样的情况：

[Info] 第 178 条数据发现超长填充: 长度 16839, 范围 [48697:65536]

如果需要准确的数据，可以联系我要准确的数据。

没有验证过关闭CP和SP是否不会报错。

KL4805

5月14日评论：

我们现在使用了 1.5B 模型进行了问题复现的时候，发现了以下的情况，下面是具体说明：

#!/bin/bash
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NPU_ASD_ENABLE=0
export WITHOUT_JIT_COMPILE=1
export HCCL_CONNECT_TIMEOUT=7200
export MOX_FILE_LARGE_FILE_TASK_NUM=2

MASTER_ADDR="localhost"
MASTER_PORT=6000
NNODES=1
NODE_RANK=0
NPUS_PER_NODE=4
WORLD_SIZE=$(($NPUS_PER_NODE * $NNODES))

TP=1
PP=1
CP=1
CP_TYPE='ulysses_cp_algo'

SEQ_LENGTH=4096
TRAIN_ITERS=100000
SAVE_ITERS=100000
MBS=1
GBS=4

SCRIPT_NAME="tune_qwen25_1point5b_test"
CKPT_LOAD_DIR="/data/work_jyl/llm_train_workdir/mg_checkpoint_dir/Qwen2.5-Coder-1.5B-Instruct-mcore-TP${TP}PP${PP}"
TOKENIZER_PATH="/data/work_jyl/llm_train_workdir/tokenizer_dir/Qwen2.5-Coder-1.5B-Instruct/"
CKPT_SAVE_DIR="/data/work_jyl/llm_train_workdir/train_output_dir/ckpt_1point5B_test/"
TENSORBOARD_PATH="/data/work_jyl/llm_train_workdir/train_output_dir/ckpt_1point5B_test/tensorboard_dir/"

DATA_PATH="/home/ma-user/work/llm_train_workdir/data_dir/4k_pack/dataset_open_tulu3/dataset_open_tulu3"

DISTRIBUTED_ARGS="
    --nproc_per_node $NPUS_PER_NODE \
    --nnodes $NNODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT
"

GPT_ARGS="
    --finetune \
    --stage sft \
    --prompt-type qwen \
    --is-instruction-dataset \
    --no-pad-to-seq-lengths \
    --tokenizer-not-use-fast \
    --use-mcore-models \
    --tensor-model-parallel-size ${TP} \
    --pipeline-model-parallel-size ${PP} \
    --context-parallel-size ${CP} \
    --context-parallel-algo ${CP_TYPE} \
    --num-layers 28 \
    --hidden-size 1536 \
    --ffn-hidden-size 8960 \
    --num-attention-heads 12 \
    --group-query-attention \
    --num-query-groups 2 \
    --tokenizer-type PretrainedFromHF \
    --tokenizer-name-or-path ${TOKENIZER_PATH} \
    --seq-length ${SEQ_LENGTH} \
    --max-position-embeddings ${SEQ_LENGTH} \
    --micro-batch-size ${MBS} \
    --global-batch-size ${GBS} \
    --make-vocab-size-divisible-by 1 \
    --padded-vocab-size 151936 \
    --rotary-base 1000000 \
    --train-iters ${TRAIN_ITERS} \
    --lr 7.5e-7 \
    --weight-decay 0.0 \
    --lr-decay-style cosine \
    --clip-grad 1.0 \
    --adam-beta1 0.9 \
    --adam-beta2 0.95 \
    --add-qkv-bias \
    --disable-bias-linear \
    --attention-dropout 0.0 \
    --init-method-std 0.02 \
    --hidden-dropout 0.0 \
    --position-embedding-type rope \
    --normalization RMSNorm \
    --norm-epsilon 1e-06 \
    --swiglu \
    --use-flash-attn \
    --use-rotary-position-embeddings \
    --no-masked-softmax-fusion \
    --attention-softmax-in-fp32 \
    --initial-loss-scale 4096 \
    --no-gradient-accumulation-fusion \
    --no-load-optim \
    --no-load-rng \
    --seed 42 \
    --bf16 \
    --sequence-parallel
"

DATA_ARGS="
    --data-path $DATA_PATH \
    --split 100,0,0 \
    --reset-attention-mask \
    --pack \
    --neat-pack
"

OUTPUT_ARGS="
    --log-interval 1 \
    --save-interval ${SAVE_ITERS} \
    --eval-interval ${SAVE_ITERS} \
    --eval-iters 0 \
    --tensorboard-dir ${TENSORBOARD_PATH}
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
    $GPT_ARGS \
    $DATA_ARGS \
    $OUTPUT_ARGS \
    --distributed-backend nccl \
    --load ${CKPT_LOAD_DIR} \
    --save ${CKPT_SAVE_DIR}

同时在 mindspeed_llm/tasks/posttrain/sft/sft_trainer.py 的 loss_func 对应信息进行简单打印验证。

def loss_func(input_tensor: torch.Tensor, output_tensor: torch.Tensor):
        """Loss function.

        Args:
            input_tensor (torch.Tensor): Used to mask out some portions of the loss
            output_tensor (torch.Tensor): The tensor with the losses
        """
        args = get_args()
        loss_mask = input_tensor

        losses = output_tensor.float()
        loss_mask = loss_mask[..., 1:].view(-1).float()

        torch.set_printoptions(threshold=float('inf'))

        # --- 开始打印 ---
        print_rank_0(f">>> [RANK 0] FULL LOSS_MASK: {loss_mask}")
        print_rank_last(f">>> [LAST RANK] FULL LOSS_MASK: {loss_mask}")

        print_rank_0(f">>> [RANK 0] LOSSES: {losses}")
        print_rank_last(f">>> [LAST RANK] LOSSES: {losses}")
        # --- 结束打印 ---

当脚本中 CP 参数设置 1 时，打印出来的部分信息如下

        [RANK 0] FULL LOSS_MASK: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
        0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
        1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,

         [RANK 0] LOSSES: tensor([[4.2512e-01, 2.5367e+00, 1.6367e+00, 1.6961e-01, 1.4108e+00, 1.3092e+00,
         8.2327e-01, 1.1029e+00, 1.4376e-01, 2.4148e+00, 6.1321e-02, 1.8374e+00,
         1.1720e+00, 1.7439e-04, 1.1598e-04, 4.1611e-01, 8.7050e-01, 2.2392e+00,
         5.0030e-02, 2.4109e-01, 1.0079e+00, 2.0378e+00, 8.3625e-01, 7.9236e-01,
         1.6209e-01, 4.0693e-02, 8.8327e-01, 7.2713e-01, 2.1057e+00, 2.8596e-03,
         1.2967e+00, 8.1886e-02, 5.5339e-01, 1.0003e+00, 3.2292e-01, 1.5620e+00,
         8.6307e-01, 4.1965e-01, 6.9908e-01, 5.4981e-01, 2.6246e-04, 4.0651e-03,
         1.1472e+00, 2.1078e+01, 1.2063e+01, 2.1832e+01, 6.5020e+00, 1.7376e+01,
         6.3529e+00, 1.4732e-01, 1.4585e-03, 6.7338e-02, 1.7311e-03, 3.4955e+00,
         2.5032e+00, 4.6733e-01, 9.0105e-03, 5.1190e-01, 6.2735e-02, 1.2426e-03,
         1.9393e-04, 3.4136e-04, 4.0330e-01, 7.7259e-01, 3.6125e-01, 1.1095e+00,
         4.1368e+00, 1.7882e+00, 2.7728e+00, 1.3014e+00, 3.1059e-01, 3.3705e+00,
         1.3529e-04, 3.0957e-02, 6.2583e-05, 3.7972e-01, 4.9615e-04, 1.0870e+00,
         1.3502e-01, 2.1065e-03, 1.3654e-01, 4.4495e-01, 6.7867e+00, 7.0889e-01,
         2.7353e+00, 4.4563e-01, 2.2313e-03, 1.2548e+00, 1.4964e+00, 1.0781e+00,
         2.9383e-01, 7.5843e-02, 6.9651e-01, 1.4363e-03, 8.4975e+00, 1.2750e-03,
         5.0008e-04, 5.9270e+00, 7.2009e-01, 1.7061e-02, 1.3338e+00, 2.0017e+00,
         4.4853e-01, 3.2517e-03, 5.3388e+00, 6.6779e-03, 1.6000e+00, 1.3463e+00,
         1.9804e-03, 6.5247e-01, 2.3823e-01, 5.5712e+00, 2.5042e+00, 4.2981e-01,

当设置 CP 参数为 2 时，对应的部分输出如下：

        [RANK 0] FULL LOSS_MASK: tensor([8.4666e-01, -2.7098e-01, 1.2439e-01, -1.7553e-01, 2.2796e-02, -5.8883e-01,
        4.6432e-01, 2.1606e-01, -7.1407e-02, -5.0121e-03, -3.0907e-01, 1.4428e-01,
        -1.8139e-01, -1.3695e-01, 4.7605e-01, 3.1797e-02, 5.4625e-02, 3.4325e-01,
        -8.8986e-02, -1.1734e-02, -2.5436e-01, 3.7254e-01, -1.1511e-01, -3.8622e-01,
        7.1654e-02, -3.7157e-01, 1.4330e-01, 7.1404e-02, 7.8611e-01, -2.1166e-01,
        -9.1914e-02, 3.8632e-02, 2.2436e-01, 2.0678e-01, -3.2763e-01, 1.2740e-02,
        -8.6300e-02, -4.4981e-02, 4.9986e-02, -1.7797e-01, 8.4349e-02, -3.1883e-01,
        -7.8243e-02, 6.9431e-01, -2.8466e-01, -1.1779e-01, 4.7424e-02, 7.0213e-01,
        9.2283e-01, 4.6044e-01, -3.1785e-01, 8.6056e-02, 7.6464e-01, -1.0241e-01,
        4.4493e-02, 2.0482e-01, 1.1438e-01, 3.7743e-01, 3.8719e-01, -2.8563e-01,
        1.2264e-03, -1.6918e-01, 3.4130e-01, 3.6473e-01, -2.0538e-02, -2.5535e-01,
        2.2680e-01, 1.6283e-01, -2.4682e-01, -8.7766e-02, 1.8041e-01, 2.0140e-01,
        5.7799e-02, 1.1291e-01, 1.5844e-01, 8.8986e-02, -2.2631e-01, -2.2729e-01,
        -2.2094e-01, -3.3153e-01, -3.2567e-01, 2.6805e-01, -2.1020e-01, -2.2143e-01,
        2.2094e-01, -8.9964e-02, 2.1801e-01, -1.7748e-01, 3.6803e-02, 4.0868e-01,
        -3.4520e-01, 7.8244e-02, -1.1047e-01, 3.8134e-01, 2.9735e-01, -3.4129e-01,

          [RANK 0] LOSSES: tensor([[9.4359e-02, -3.2176e-01, -6.2206e-01, -2.8466e-01, -3.6962e-01, -3.0615e-01,
         -6.8478e-02, -1.9164e-01, -6.8065e-01, -2.3461e-01, 2.2826e-01, -4.9376e-02,
         -1.0486e-01, 4.0965e-01, 5.5955e-01, -1.0269e-02, 6.1228e-01, 1.6381e-01,
         -1.7309e-01, -7.4091e-02, 4.6824e-01, -2.7587e-01, -4.9498e-02, 5.5173e-01,
         -5.6457e-02, 3.9733e-02, -1.6284e-01, -5.3806e-01, 4.1454e-01, 7.3920e-01,
         1.1653e-03, 3.4227e-01, 5.5114e-02, 2.3315e-01, -4.3406e-01, -1.5307e-01,
         -2.7001e-01, 5.6225e-03, -5.4978e-01, 4.8398e-02, 2.6121e-01, -4.4005e-02,
         -1.7065e-01, 4.9252e-02, -2.5237e-02, -3.2286e-02, 1.5258e-01, 5.3415e-01,
         -4.6922e-01, 3.0028e-01, -5.9386e-02, -3.3349e-01, -1.2573e-01, -4.9755e-01,
         1.8139e-01, 1.2621e-01, 5.3160e-02, -5.4588e-01, 2.3071e-01, -4.1747e-01,
         1.2341e-01, 1.3207e-01, 1.0559e-01, -7.8274e-03, -4.5458e-01, -2.5339e-01,
         -1.4672e-01, 8.4276e-01, 7.5559e-02, 9.7042e-02, 7.5315e-02, -4.7667e-02,
         -3.2177e-01, 1.1398e-02, 4.6445e-02, 1.5502e-01, -1.3452e-01, 3.8878e-02,
         2.1264e-01, 3.2773e-02, 5.2438e-01, -1.4999e-02, 2.5926e-01, 2.5829e-01,
         8.5252e-01, -7.6534e-02, -1.0900e-01, -2.2338e-01, 1.5502e-01, -3.4814e-01,
         -4.5653e-01, 3.2957e-01, 1.7846e-01, 7.7267e-02, 2.5176e-02, -1.1245e-02,
         -1.3891e-01, 9.1426e-02, -2.8660e-01, -1.8627e-01, -4.7055e-02, -1.8530e-01,
         1.5599e-01, 4.8973e-01, 6.1583e-02, -3.7290e-02, -1.3696e-01, 1.8530e-01,

问题出在 CP = 2 时，为什么打印出来的 loss_mask 并不是 0/1 的形式？这是正常的吗？并且如果开启 CP = 2，会直接导致 loss_func 函数进入 args.check_for_nan_in_loss_and_grad 分支直接抛异常，请问这是什么情况？

yanzhixiao成员

5月18日评论：

完整运行脚本（被训练模型Qwen3-Coder-30B-A3B-Instruct）

#!/bin/bash
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NPU_ASD_ENABLE=0
export WITHOUT_JIT_COMPILE=1
export HCCL_CONNECT_TIMEOUT=7200
export MOX_FILE_LARGE_FILE_TASK_NUM=2

MASTER_HOST="$VC_WORKER_HOSTS"
MASTER_ADDR="${VC_WORKER_HOSTS%%,*}"
MASTER_PORT=6000
JOB_ID="1234"
NNODES="$MA_NUM_HOSTS"
NODE_RANK="$VC_TASK_INDEX"
NPUS_PER_NODE="$MA_NUM_GPUS"
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

if [[ $NODE_RANK == 0 ]]; then
   EXT_ARGS="--rdzv_conf=is_host=1"
else
   EXT_ARGS=""
fi

TP=4
PP=4
EP=8
CP=2
CP_TYPE='ulysses_cp_algo'

SEQ_LENGTH=65536
TRAIN_ITERS=2500
SAVE_ITERS=300
MBS=1
GBS=16

SCRIPT_NAME="tune_qwen3_30b_a3b_base_tulu3"

CKPT_LOAD_DIR="/mnt/efs/mg_checkpoint_dir/Qwen3-Coder-30B-A3B-Instruct-mcore-TP${TP}PP${PP}EP${EP}"
TOKENIZER_PATH="/mnt/efs/tokenizer_dir/Qwen3-Coder-30B-A3B-Instruct/"
CKPT_SAVE_DIR="/home/ma-user/modelarts/outputs/train_output_0/"
TENSORBOARD_PATH="/home/ma-user/modelarts/outputs/train_output_0/tensorboard_dir/"
LOG_PATH="/mnt/efs/jyl/logs/${SCRIPT_NAME}.log"

# total 9110
# GBS 32, 285 iters / epoch
# GBS 16, 570 iters / epoch
DATA_PATH="179 /mnt/efs/jyl/data_dir/sft_data/64k_pack/dataset_open_tulu3/dataset_open_tulu3"
# DATA_PATH+=" 5129 /mnt/efs/jyl/data_dir/sft_data/64k_pack/dataset_open_lmsys_gpt_oss_sft_en_droplast/dataset_open_lmsys_gpt_oss_sft_en_droplast"


DISTRIBUTED_ARGS="
   --nproc_per_node $NPUS_PER_NODE \
   --nnodes $NNODES \
   --node_rank $NODE_RANK \
   --master_addr $MASTER_ADDR \
   --master_port $MASTER_PORT
"

MOE_ARGS="
   --num-experts 128 \
   --expert-tensor-parallel-size 1 \
   --moe-router-topk 8 \
   --moe-ffn-hidden-size 768 \
   --moe-grouped-gemm \
   --moe-permutation-async-comm \
   --moe-permute-fusion \
   --moe-alltoall-overlap-comm \
   --moe-token-dispatcher-type alltoall \
   --moe-router-load-balancing-type aux_loss \
   --moe-layer-freq -1 \
   --first-k-dense-replace -1 \
   --moe-aux-loss-coeff 0.001
"

OPTIMIZE_ARGS="
   --use-flash-attn \
   --use-fused-rotary-pos-emb \
   --sequence-parallel \
   --use-rotary-position-embeddings \
   --use-fused-swiglu \
   --use-fused-rmsnorm \
   --no-masked-softmax-fusion \
   --use-distributed-optimizer \
   --swap-optimizer \
   --swap-attention \
   --gemm-gradient-accumulation-fusion \
   --recompute-granularity full \
   --recompute-method uniform \
   --recompute-num-layers 1
"

TRAIN_ARGS="
   --reset-attention-mask \
   --attention-mask-type general \
   --micro-batch-size ${MBS} \
   --global-batch-size ${GBS} \
   --lr 5e-7 \
   --lr-decay-style cosine \
   --min-lr 5e-8 \
   --weight-decay 1e-1 \
   --lr-warmup-fraction 0.03 \
   --attention-dropout 0.0 \
   --init-method-std 0.01 \
   --hidden-dropout 0.0 \
   --clip-grad 1.0 \
   --adam-beta1 0.9 \
   --adam-beta2 0.95 \
   --initial-loss-scale 4096 \
   --seed 42 \
   --bf16 \
   --train-iters ${TRAIN_ITERS} \
   --seq-length ${SEQ_LENGTH} \
   --manual-gc \
   --manual-gc-interval 50
"

MODEL_PARALLEL_ARGS="
   --tensor-model-parallel-size ${TP} \
   --pipeline-model-parallel-size ${PP} \
   --expert-model-parallel-size ${EP} \
   --context-parallel-size ${CP} \
   --context-parallel-algo ${CP_TYPE}
"

GPT_ARGS="
   --use-mcore-models \
   --spec mindspeed_llm.tasks.models.spec.qwen3_spec layer_spec \
   --kv-channels 128 \
   --qk-layernorm \
   --norm-topk-prob \
   --tokenizer-name-or-path ${TOKENIZER_PATH} \
   --max-position-embeddings ${SEQ_LENGTH} \
   --num-layers 48 \
   --hidden-size 2048 \
   --ffn-hidden-size 6144 \
   --num-attention-heads 32 \
   --tokenizer-type PretrainedFromHF \
   --make-vocab-size-divisible-by 1 \
   --padded-vocab-size 151936 \
   --rotary-base 1000000 \
   --untie-embeddings-and-output-weights \
   --disable-bias-linear \
   --position-embedding-type rope \
   --normalization RMSNorm \
   --swiglu \
   --attention-softmax-in-fp32 \
   --no-gradient-accumulation-fusion \
   --group-query-attention \
   --num-query-groups 4 \
   --tensorboard-dir ${TENSORBOARD_PATH} \
   --tensorboard-log-interval 10
"

DATA_ARGS="
   --data-path $DATA_PATH \
   --split 100,0,0
"

OUTPUT_ARGS="
   --log-interval 1 \
   --save-interval ${SAVE_ITERS} \
   --eval-interval ${SAVE_ITERS} \
   --eval-iters 0 \
   --no-load-optim \
   --load ${CKPT_LOAD_DIR} \
   --save ${CKPT_SAVE_DIR} \
   --no-load-rng \
   --log-throughput
"

TUNE_ARGS="
   --finetune \
   --stage sft \
   --is-instruction-dataset \
   --tokenizer-not-use-fast \
   --prompt-type qwen3 \
   --neat-pack \
   --pack \
   --reset-position-ids
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
   $TUNE_ARGS \
   $GPT_ARGS \
   $DATA_ARGS \
   $MOE_ARGS \
   $OUTPUT_ARGS \
   $OPTIMIZE_ARGS \
   $TRAIN_ARGS \
   $MODEL_PARALLEL_ARGS \
   --distributed-backend nccl \
   | tee ${LOG_PATH}

环境信息
MindSpeed-LLM 2.3.0, MindSpeed0.12.1，Megatron参考的2.3.0的安装要求切换的分支。
torch_npu 2.7.1post2，CANN版本 8.3.RC1
数据
数据来自于tulu3-instruction-following: https://huggingface.co/datasets/allenai/tulu-3-sft-personas-instruction-following
预处理使用的选项是：alpaca handler，neat-pack, pack均打开，上下文长度65536. prompt-type Qwen3

我无法给出具体的保存信息，因为modelarts平台没有给具体的报错信息，只有这个：


W0510 09:24:21.680845 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1171 closing signal SIGTERM
W0510 09:24:21.693475 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1172 closing signal SIGTERM
W0510 09:24:21.696840 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1173 closing signal SIGTERM
W0510 09:24:21.699285 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1174 closing signal SIGTERM
W0510 09:24:21.701289 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1175 closing signal SIGTERM
W0510 09:24:21.719487 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1177 closing signal SIGTERM
W0510 09:24:21.720776 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:900] Sending process 1178 closing signal SIGTERM
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
[ERROR] TBE Subprocess[task_distribute] raise error[], main process disappeared!
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
E0510 09:24:30.336779 949 site-packages/torch/distributed/elastic/multiprocessing/api.py:874] failed (exitcode: -11) local_rank: 5 (pid: 1176) of binary: /home/ma-user/miniconda3/envs/mindspeed-230-new/bin/python3.10
Traceback (most recent call last):
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/bin/torchrun", line 6, in <module>
   sys.exit(main())
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 355, in wrapper
   return f(*args, **kwargs)
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/run.py", line 892, in main
   run(args)
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/run.py", line 883, in run
   elastic_launch(
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 139, in __call__
   return launch_agent(self._config, self._entrypoint, list(args))
 File "/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/site-packages/torch/distributed/launcher/api.py", line 270, in launch_agent
   raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError: 
============================================================
posttrain_gpt.py FAILED
------------------------------------------------------------
Failures:
 <NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
 time      : 2026-05-10_09:24:21
 host      : ma-job-555a0749-cfed-45b6-84b1-29d7897832c5-worker-0.ma-job-555a0749-cfed-45b6-84b1-29d7897832c5.os-d910b-dev-codemate.svc.cluster.local
 rank      : 5 (local_rank: 5)
 exitcode  : -11 (pid: 1176)
 error_file: <N/A>
 traceback : Signal 11 (SIGSEGV) received by PID 1176
============================================================
/home/ma-user/miniconda3/envs/mindspeed-230-new/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 30 leaked semaphore objects to clean up at shutdown
 warnings.warn('resource_tracker: There appear to be %d '
[ERROR] 2026-05-10-09:24:30 (PID:949, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

训练配置是64卡910B3.

关于这条数据的特征，因为有点长，我无法给出具体的分布，但是有这样的情况：

[Info] 第 178 条数据发现超长填充: 长度 16839, 范围 [48697:65536]

如果需要准确的数据，可以联系我要准确的数据。

没有验证过关闭CP和SP是否不会报错。

@PH-BUF

感谢您的反馈，经过您提供的信息，我将一一解答您的问题：
1.首先sft的loss func的计算不会触发除零，正如您给出的例子：TP4CP2，同时开启sequence-parallel，在进入loss func计算之前的output layer就已经完成了序列维度上的TP/SP聚合，此时参与loss func计算的loss mask只是经过了CP切分，所以在该函数中只考虑了CP域的聚合。总的来说，TP/SP域已经在进入loss func函数完成聚合，loss func只需考虑CP域all_reduce，参与计算的loss mask是完整序列长度，不会触发除零错误。
2.关于您训练过程中出现NaN问题，我建议你检查您的数据处理是否存在问题，因为我们仓库目前不支持tulu3-instruction-following数据集的处理，您是否自己完成了适配处理。或者您可以尝试使用我们仓库指定数据集验证是否依旧存在NaN问题。

yanzhixiao成员

5月18日评论：

我们现在使用了 1.5B 模型进行了问题复现的时候，发现了以下的情况，下面是具体说明：

#!/bin/bash
export CUDA_DEVICE_MAX_CONNECTIONS=1
export NPU_ASD_ENABLE=0
export WITHOUT_JIT_COMPILE=1
export HCCL_CONNECT_TIMEOUT=7200
export MOX_FILE_LARGE_FILE_TASK_NUM=2

MASTER_ADDR="localhost"
MASTER_PORT=6000
NNODES=1
NODE_RANK=0
NPUS_PER_NODE=4
WORLD_SIZE=$(($NPUS_PER_NODE * $NNODES))

TP=1
PP=1
CP=1
CP_TYPE='ulysses_cp_algo'

SEQ_LENGTH=4096
TRAIN_ITERS=100000
SAVE_ITERS=100000
MBS=1
GBS=4

SCRIPT_NAME="tune_qwen25_1point5b_test"
CKPT_LOAD_DIR="/data/work_jyl/llm_train_workdir/mg_checkpoint_dir/Qwen2.5-Coder-1.5B-Instruct-mcore-TP${TP}PP${PP}"
TOKENIZER_PATH="/data/work_jyl/llm_train_workdir/tokenizer_dir/Qwen2.5-Coder-1.5B-Instruct/"
CKPT_SAVE_DIR="/data/work_jyl/llm_train_workdir/train_output_dir/ckpt_1point5B_test/"
TENSORBOARD_PATH="/data/work_jyl/llm_train_workdir/train_output_dir/ckpt_1point5B_test/tensorboard_dir/"

DATA_PATH="/home/ma-user/work/llm_train_workdir/data_dir/4k_pack/dataset_open_tulu3/dataset_open_tulu3"

DISTRIBUTED_ARGS="
   --nproc_per_node $NPUS_PER_NODE \
   --nnodes $NNODES \
   --node_rank $NODE_RANK \
   --master_addr $MASTER_ADDR \
   --master_port $MASTER_PORT
"

GPT_ARGS="
   --finetune \
   --stage sft \
   --prompt-type qwen \
   --is-instruction-dataset \
   --no-pad-to-seq-lengths \
   --tokenizer-not-use-fast \
   --use-mcore-models \
   --tensor-model-parallel-size ${TP} \
   --pipeline-model-parallel-size ${PP} \
   --context-parallel-size ${CP} \
   --context-parallel-algo ${CP_TYPE} \
   --num-layers 28 \
   --hidden-size 1536 \
   --ffn-hidden-size 8960 \
   --num-attention-heads 12 \
   --group-query-attention \
   --num-query-groups 2 \
   --tokenizer-type PretrainedFromHF \
   --tokenizer-name-or-path ${TOKENIZER_PATH} \
   --seq-length ${SEQ_LENGTH} \
   --max-position-embeddings ${SEQ_LENGTH} \
   --micro-batch-size ${MBS} \
   --global-batch-size ${GBS} \
   --make-vocab-size-divisible-by 1 \
   --padded-vocab-size 151936 \
   --rotary-base 1000000 \
   --train-iters ${TRAIN_ITERS} \
   --lr 7.5e-7 \
   --weight-decay 0.0 \
   --lr-decay-style cosine \
   --clip-grad 1.0 \
   --adam-beta1 0.9 \
   --adam-beta2 0.95 \
   --add-qkv-bias \
   --disable-bias-linear \
   --attention-dropout 0.0 \
   --init-method-std 0.02 \
   --hidden-dropout 0.0 \
   --position-embedding-type rope \
   --normalization RMSNorm \
   --norm-epsilon 1e-06 \
   --swiglu \
   --use-flash-attn \
   --use-rotary-position-embeddings \
   --no-masked-softmax-fusion \
   --attention-softmax-in-fp32 \
   --initial-loss-scale 4096 \
   --no-gradient-accumulation-fusion \
   --no-load-optim \
   --no-load-rng \
   --seed 42 \
   --bf16 \
   --sequence-parallel
"

DATA_ARGS="
   --data-path $DATA_PATH \
   --split 100,0,0 \
   --reset-attention-mask \
   --pack \
   --neat-pack
"

OUTPUT_ARGS="
   --log-interval 1 \
   --save-interval ${SAVE_ITERS} \
   --eval-interval ${SAVE_ITERS} \
   --eval-iters 0 \
   --tensorboard-dir ${TENSORBOARD_PATH}
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py \
   $GPT_ARGS \
   $DATA_ARGS \
   $OUTPUT_ARGS \
   --distributed-backend nccl \
   --load ${CKPT_LOAD_DIR} \
   --save ${CKPT_SAVE_DIR}

同时在 mindspeed_llm/tasks/posttrain/sft/sft_trainer.py 的 loss_func 对应信息进行简单打印验证。

def loss_func(input_tensor: torch.Tensor, output_tensor: torch.Tensor):
       """Loss function.

       Args:
           input_tensor (torch.Tensor): Used to mask out some portions of the loss
           output_tensor (torch.Tensor): The tensor with the losses
       """
       args = get_args()
       loss_mask = input_tensor

       losses = output_tensor.float()
       loss_mask = loss_mask[..., 1:].view(-1).float()

       torch.set_printoptions(threshold=float('inf'))

       # --- 开始打印 ---
       print_rank_0(f">>> [RANK 0] FULL LOSS_MASK: {loss_mask}")
       print_rank_last(f">>> [LAST RANK] FULL LOSS_MASK: {loss_mask}")

       print_rank_0(f">>> [RANK 0] LOSSES: {losses}")
       print_rank_last(f">>> [LAST RANK] LOSSES: {losses}")
       # --- 结束打印 ---

当脚本中 CP 参数设置 1 时，打印出来的部分信息如下

       [RANK 0] FULL LOSS_MASK: tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,
       0., 0., 0., 0., 0., 0., 0., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,
       1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,

        [RANK 0] LOSSES: tensor([[4.2512e-01, 2.5367e+00, 1.6367e+00, 1.6961e-01, 1.4108e+00, 1.3092e+00,
        8.2327e-01, 1.1029e+00, 1.4376e-01, 2.4148e+00, 6.1321e-02, 1.8374e+00,
        1.1720e+00, 1.7439e-04, 1.1598e-04, 4.1611e-01, 8.7050e-01, 2.2392e+00,
        5.0030e-02, 2.4109e-01, 1.0079e+00, 2.0378e+00, 8.3625e-01, 7.9236e-01,
        1.6209e-01, 4.0693e-02, 8.8327e-01, 7.2713e-01, 2.1057e+00, 2.8596e-03,
        1.2967e+00, 8.1886e-02, 5.5339e-01, 1.0003e+00, 3.2292e-01, 1.5620e+00,
        8.6307e-01, 4.1965e-01, 6.9908e-01, 5.4981e-01, 2.6246e-04, 4.0651e-03,
        1.1472e+00, 2.1078e+01, 1.2063e+01, 2.1832e+01, 6.5020e+00, 1.7376e+01,
        6.3529e+00, 1.4732e-01, 1.4585e-03, 6.7338e-02, 1.7311e-03, 3.4955e+00,
        2.5032e+00, 4.6733e-01, 9.0105e-03, 5.1190e-01, 6.2735e-02, 1.2426e-03,
        1.9393e-04, 3.4136e-04, 4.0330e-01, 7.7259e-01, 3.6125e-01, 1.1095e+00,
        4.1368e+00, 1.7882e+00, 2.7728e+00, 1.3014e+00, 3.1059e-01, 3.3705e+00,
        1.3529e-04, 3.0957e-02, 6.2583e-05, 3.7972e-01, 4.9615e-04, 1.0870e+00,
        1.3502e-01, 2.1065e-03, 1.3654e-01, 4.4495e-01, 6.7867e+00, 7.0889e-01,
        2.7353e+00, 4.4563e-01, 2.2313e-03, 1.2548e+00, 1.4964e+00, 1.0781e+00,
        2.9383e-01, 7.5843e-02, 6.9651e-01, 1.4363e-03, 8.4975e+00, 1.2750e-03,
        5.0008e-04, 5.9270e+00, 7.2009e-01, 1.7061e-02, 1.3338e+00, 2.0017e+00,
        4.4853e-01, 3.2517e-03, 5.3388e+00, 6.6779e-03, 1.6000e+00, 1.3463e+00,
        1.9804e-03, 6.5247e-01, 2.3823e-01, 5.5712e+00, 2.5042e+00, 4.2981e-01,

当设置 CP 参数为 2 时，对应的部分输出如下：

       [RANK 0] FULL LOSS_MASK: tensor([8.4666e-01, -2.7098e-01, 1.2439e-01, -1.7553e-01, 2.2796e-02, -5.8883e-01,
       4.6432e-01, 2.1606e-01, -7.1407e-02, -5.0121e-03, -3.0907e-01, 1.4428e-01,
       -1.8139e-01, -1.3695e-01, 4.7605e-01, 3.1797e-02, 5.4625e-02, 3.4325e-01,
       -8.8986e-02, -1.1734e-02, -2.5436e-01, 3.7254e-01, -1.1511e-01, -3.8622e-01,
       7.1654e-02, -3.7157e-01, 1.4330e-01, 7.1404e-02, 7.8611e-01, -2.1166e-01,
       -9.1914e-02, 3.8632e-02, 2.2436e-01, 2.0678e-01, -3.2763e-01, 1.2740e-02,
       -8.6300e-02, -4.4981e-02, 4.9986e-02, -1.7797e-01, 8.4349e-02, -3.1883e-01,
       -7.8243e-02, 6.9431e-01, -2.8466e-01, -1.1779e-01, 4.7424e-02, 7.0213e-01,
       9.2283e-01, 4.6044e-01, -3.1785e-01, 8.6056e-02, 7.6464e-01, -1.0241e-01,
       4.4493e-02, 2.0482e-01, 1.1438e-01, 3.7743e-01, 3.8719e-01, -2.8563e-01,
       1.2264e-03, -1.6918e-01, 3.4130e-01, 3.6473e-01, -2.0538e-02, -2.5535e-01,
       2.2680e-01, 1.6283e-01, -2.4682e-01, -8.7766e-02, 1.8041e-01, 2.0140e-01,
       5.7799e-02, 1.1291e-01, 1.5844e-01, 8.8986e-02, -2.2631e-01, -2.2729e-01,
       -2.2094e-01, -3.3153e-01, -3.2567e-01, 2.6805e-01, -2.1020e-01, -2.2143e-01,
       2.2094e-01, -8.9964e-02, 2.1801e-01, -1.7748e-01, 3.6803e-02, 4.0868e-01,
       -3.4520e-01, 7.8244e-02, -1.1047e-01, 3.8134e-01, 2.9735e-01, -3.4129e-01,

         [RANK 0] LOSSES: tensor([[9.4359e-02, -3.2176e-01, -6.2206e-01, -2.8466e-01, -3.6962e-01, -3.0615e-01,
        -6.8478e-02, -1.9164e-01, -6.8065e-01, -2.3461e-01, 2.2826e-01, -4.9376e-02,
        -1.0486e-01, 4.0965e-01, 5.5955e-01, -1.0269e-02, 6.1228e-01, 1.6381e-01,
        -1.7309e-01, -7.4091e-02, 4.6824e-01, -2.7587e-01, -4.9498e-02, 5.5173e-01,
        -5.6457e-02, 3.9733e-02, -1.6284e-01, -5.3806e-01, 4.1454e-01, 7.3920e-01,
        1.1653e-03, 3.4227e-01, 5.5114e-02, 2.3315e-01, -4.3406e-01, -1.5307e-01,
        -2.7001e-01, 5.6225e-03, -5.4978e-01, 4.8398e-02, 2.6121e-01, -4.4005e-02,
        -1.7065e-01, 4.9252e-02, -2.5237e-02, -3.2286e-02, 1.5258e-01, 5.3415e-01,
        -4.6922e-01, 3.0028e-01, -5.9386e-02, -3.3349e-01, -1.2573e-01, -4.9755e-01,
        1.8139e-01, 1.2621e-01, 5.3160e-02, -5.4588e-01, 2.3071e-01, -4.1747e-01,
        1.2341e-01, 1.3207e-01, 1.0559e-01, -7.8274e-03, -4.5458e-01, -2.5339e-01,
        -1.4672e-01, 8.4276e-01, 7.5559e-02, 9.7042e-02, 7.5315e-02, -4.7667e-02,
        -3.2177e-01, 1.1398e-02, 4.6445e-02, 1.5502e-01, -1.3452e-01, 3.8878e-02,
        2.1264e-01, 3.2773e-02, 5.2438e-01, -1.4999e-02, 2.5926e-01, 2.5829e-01,
        8.5252e-01, -7.6534e-02, -1.0900e-01, -2.2338e-01, 1.5502e-01, -3.4814e-01,
        -4.5653e-01, 3.2957e-01, 1.7846e-01, 7.7267e-02, 2.5176e-02, -1.1245e-02,
        -1.3891e-01, 9.1426e-02, -2.8660e-01, -1.8627e-01, -4.7055e-02, -1.8530e-01,
        1.5599e-01, 4.8973e-01, 6.1583e-02, -3.7290e-02, -1.3696e-01, 1.8530e-01,

@PH-BUF

感谢您的反馈，针对您使用 1.5B 模型进行了问题复现时，发现开启CP时loss mask为浮点数的问题，我尝试复现但未曾成功。不知道您是否方便能提供改脚本数据和tokenizer方便我复现定位问题。除此之外，我将给您以下建议：
1.关闭脚本中--no-pad-to-seq-lengths特性，理论上pack与动态序列是两种不同的训练场景，不建议一起使用。
2.loss mask主要来自数据处理的结果，开启CP时只会使用get_batch_on_this_cp_rank对数据进行切分，您可以重点排查SFTTrainer中的get_batch函数

希望我的建议可以帮助到您，如果您有其他问题欢迎反馈交流!

HANHU1CHEN成员

6月1日评论：

您好，由于此issue pending时间过长（超过两周），我们没有接收到您更进一步的反馈，因此我们将根据流程关闭此 issue。
如后续有新的疑问或补充内容，欢迎随时重新打开本 issue，或另起一个新的 issue 进行提问。
感谢您的理解与支持！

H HANHU1CHEN成员

6月1日添加了label：resolved；删除了label：pending

H HANHU1CHEN成员

6月1日 issue状态由 TODO 改变为 DONE

H HANHU1CHEN成员

6月1日关闭了 issue