Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

mindspeed-llm 2.2.0.
选用3k数据训练，loss正常（起始loss小于1）；将序列长度扩展到6k后 loss爆炸(起始loss大于5，grad norm 40+)。数据在其他模型上验证过，没有问题。

尝试过多种重计算和CP方案，loss都不正确。

TP=1
PP=8
EP=16

CP=4

SEQ_LEN=6144

DISTRIBUTED_ARGS="
--nproc_per_node $NPUS_PER_NODE
--nnodes $NODE_NUM
--node_rank $RANK
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT
"

--moe-zero-memory level1 \

MOE_ARGS="
--moe-grouped-gemm
--moe-alltoall-overlap-comm
--moe-permutation-async-comm
--moe-token-dispatcher-type alltoall_seq
--first-k-dense-replace 1
--moe-layer-freq 1
--n-shared-experts 1
--num-experts 128
--moe-router-topk 8
--moe-ffn-hidden-size 1408
--moe-router-load-balancing-type none
--moe-router-num-groups 1
--moe-router-group-topk 1
--norm-topk-prob
--moe-router-score-function sigmoid
--moe-router-enable-expert-bias
--moe-router-dtype fp32
"

# --recompute-method uniform \
# --recompute-granularity full \
# --recompute-num-layers 1 \

# --moe-zero-memory level1 \
# --recompute-activation-function \

# --context-parallel-size ${CP} \
# --use-cp-send-recv-overlap \
# --attention-mask-type general \
# --cp-attention-mask-type general \
# --ulysses-degree-in-cp 2 \
# --context-parallel-algo hybrid_cp_algo \

GPT_ARGS="
--recompute-method uniform
--recompute-granularity full
--recompute-num-layers 1
--tensor-model-parallel-size ${TP}
--pipeline-model-parallel-size ${PP}
--expert-model-parallel-size ${EP}
--noop-layers 1,47
--sequence-parallel
--use-mcore-models
--use-flash-attn
--post-norm
--kv-channels 128
--use-fused-rmsnorm
--use-fused-swiglu
--overlap-grad-reduce
--use-distributed-optimizer
--num-layers 48
--hidden-size 4096
--ffn-hidden-size 10944
--num-attention-heads 96
--seq-length ${SEQ_LEN}
--micro-batch-size 1
--global-batch-size 64
--max-position-embeddings 131072
--padded-vocab-size 151552
--make-vocab-size-divisible-by 1
--group-query-attention
--num-query-groups 8
--disable-bias-linear
--add-qkv-bias
--position-embedding-type rope
--rotary-percent 0.5
--no-rope-fusion
--normalization RMSNorm
--swiglu
--tokenizer-type PretrainedFromHF
--tokenizer-name-or-path ${TOKENIZER_PATH}
--lr 5e-6
--norm-epsilon 1e-05
--train-iters 100
--lr-decay-style cosine
--untie-embeddings-and-output-weights
--attention-dropout 0.0
--init-method-std 0.01
--hidden-dropout 0.0
--no-masked-softmax-fusion
--attention-softmax-in-fp32
--min-lr 1.25e-7
--weight-decay 1e-1
--lr-warmup-fraction 0.01
--clip-grad 1.0
--adam-beta1 0.9
--initial-loss-scale 4096
--adam-beta2 0.95
--no-load-optim
--no-load-rng
--no-save-optim
--no-save-rng
--no-shared-storage
--no-gradient-accumulation-fusion
--no-bias-swiglu-fusion
--bf16
"

SFT_ARGS="
--finetune
--stage sft
--is-instruction-dataset
--prompt-type empty
"

DATA_ARGS="
--data-path ${DATA_PATH}/glm45
--split 100,0,0
"

OUTPUT_ARGS="
--log-interval 1
--save-interval 10
--eval-interval 20000
--eval-iters 0
"

我做数据的时候拼好了prompt，所以训练时template采用的是empty。

Environment / 环境信息 (Mandatory / 必填)

mindspeed-llm 2.2.0
CANN 8.3.RC1 (其余环境均使用CANN配套版本)

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

根据配置文件拉起训练即可复现

Describe the expected behavior / 预期结果 (Mandatory / 必填)

期望loss正常，grad_norm正常

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

[2025-12-17 08:33:45] iteration 1/ 100 | consumed samples: 64 | elapsed time per iteration (ms): 427079.3 | learning rate: 5.000000E-06 | global batch size: 64 | lm loss: 7.680722E+00 | loss scale: 1.0 | grad norm: 81.220 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-12-17 08:34:55] iteration 2/ 100 | consumed samples: 128 | elapsed time per iteration (ms): 69876.8 | learning rate: 4.998773E-06 | global batch size: 64 | lm loss: 6.719289E+00 | loss scale: 1.0 | grad norm: 247.244 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-12-17 08:36:05] iteration 3/ 100 | consumed samples: 192 | elapsed time per iteration (ms): 70662.6 | learning rate: 4.995093E-06 | global batch size: 64 | lm loss: 6.511322E+00 | loss scale: 1.0 | grad norm: 61.512 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-12-17 08:37:17] iteration 4/ 100 | consumed samples: 256 | elapsed time per iteration (ms): 71810.8 | learning rate: 4.988963E-06 | global batch size: 64 | lm loss: 5.613183E+00 | loss scale: 1.0 | grad norm: 33.555 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[2025-12-17 08:38:29] iteration 5/ 100 | consumed samples: 320 | elapsed time per iteration (ms): 72076.1 | learning rate: 4.980390E-06 | global batch size: 64 | lm loss: 5.123504E+00 | loss scale: 1.0 | grad norm: 31.639 | num zeros: 0 | number of skipped iterations: 0 | number of nan iterations: 0 |

Special notes for this issue/备注 (Optional / 选填)