Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

目前Qwen3VL的微调代码无法保存模型，训完了就是训完了，没保存，手动加上保存的选项会报错（参照qwen2.5vl的保存选项）。

Environment / 环境信息 (Mandatory / 必填)

910b2，torch2.7.1，torch_npu2.7.1rc1,cann 8.2.rc1

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

直接跑一个iter，在output_args那里加一个savepath，就会报错，无法保存。sh代码如下

`#!/bin/bash

source /usr/local/Ascend/ascend-toolkit/set_env.sh

该变量只用于规避megatron对其校验，对npu无效

export CUDA_DEVICE_MAX_CONNECTIONS=2 # 开启FSDP2时，不能置为1
export ASCEND_SLOG_PRINT_TO_STDOUT=0
export ASCEND_GLOBAL_LOG_LEVEL=3
export TASK_QUEUE_ENABLE=2
export COMBINED_ENABLE=1
export CPU_AFFINITY_CONF=1
export HCCL_CONNECT_TIMEOUT=7200
export NPU_ASD_ENABLE=0
export ASCEND_LAUNCH_BLOCKING=0
export ACLNN_CACHE_LIMIT=100000
export TOKENIZERS_PARALLELISM=false
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

export HCCL_SOCKET_IFNAME=

export GLOO_SOCKET_IFNAME=

export ASCEND_RT_VISIBLE_DEVICES=

NPUS_PER_NODE=2
MASTER_ADDR=localhost
MASTER_PORT=6000
NNODES=1
NODE_RANK=0
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

MM_DATA="./examples/qwen3vl/data.json"
MM_MODEL="./examples/qwen3vl/model.json"
MM_TOOL="./mindspeed_mm/tools/tools.json"
LOAD_PATH="ckpt/Qwen3-VL-30B-A3B-Thinking"
SAVE_PATH="ckpt/Qwen3-VL-30B-A3B-Thinking_health_sft_"
FSDP2_PATH="./examples/qwen3vl/fsdp2_config.yaml"

TP=1
PP=1
CP=2
MBS=1
GRAD_ACC_STEP=1
SEQ_LEN=1024
DP=$(($WORLD_SIZE/$TP/$PP/$CP)) GBS=$(($MBS*$GRAD_ACC_STEP*$DP))

DISTRIBUTED_ARGS="
--nproc_per_node $NPUS_PER_NODE
--nnodes $NNODES
--node_rank $NODE_RANK
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT
"

GPT_ARGS中模型相关参数具体配置在example/qwen2vl/model_xb.json中，训练相关参数配置在这里

GPT_ARGS="
--use-mcore-models
--tensor-model-parallel-size ${TP}
--pipeline-model-parallel-size ${PP}
--context-parallel-size ${CP}
--context-parallel-algo ulysses_cp_algo
--micro-batch-size ${MBS}
--global-batch-size ${GBS}
--tokenizer-type NullTokenizer
--vocab-size 152064
--seq-length ${SEQ_LEN}
--make-vocab-size-divisible-by 1
--normalization RMSNorm
--use-fused-rmsnorm
--swiglu
--use-fused-swiglu
--no-masked-softmax-fusion
--lr 1.0e-5
--lr-decay-style cosine
--weight-decay 0
--train-iters 1
--lr-warmup-fraction 0.1
--clip-grad 0.0
--adam-beta1 0.9
--adam-beta2 0.999
--no-gradient-accumulation-fusion
--seed 42
--load $LOAD_PATH
--use-flash-attn
--no-load-optim
--no-load-rng
--use-torch-fsdp2
--ckpt-format torch_dcp
--fsdp2-config-path $FSDP2_PATH
--no-save-optim
--no-save-rng
--num-workers 8
--untie-embeddings-and-output-weights
--optimizer-selection fused_torch_adamw
--use-cpu-initialization
"

--use-torch-fsdp2 \

--ckpt-format torch_dcp \

--fsdp2-config-path $FSDP2_PATH \

--use-torch-fsdp2 \

MM_ARGS="
--mm-data $MM_DATA
--mm-model $MM_MODEL
--mm-tool $MM_TOOL
"

OUTPUT_ARGS="
--log-interval 1
--save-interval 10000
--eval-interval 10000
--eval-iters 5000
--save SAVE_PATH \ " logfile=(date +%Y%m%d)$(date +%H%M%S)
mkdir -p logs
torchrun $DISTRIBUTED_ARGS pretrain_transformers.py
$GPT_ARGS
$MM_ARGS
$OUTPUT_ARGS
--distributed-backend nccl
--distributed-timeout-minutes 45
2>&1 | tee logs/train${logfile}.log
chmod 440 logs/train_${logfile}.log
find $SAVE_PATH -type d -exec chmod 750 {} ;
find $SAVE_PATH -type f -exec chmod 640 {} ;`

Describe the expected behavior / 预期结果 (Mandatory / 必填)

应该可以保存才对

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

[2025-10-23 20:17:46] iteration 1/ 1 | consumed samples: 1 | elapsed time per iteration (ms): 48999.3 | learning rate: 0.000000E+00 | global batch size: 1 | loss: 2.140177E+00 | loss scale: 1.0 | grad norm: 44.250 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 |
[after training is done] datetime: 2025-10-23 20:17:46
saving checkpoint at iteration 1 to ckpt/Qwen3-VL-30B-A3B-Thinking_health_sft_ in torch_dcp format
/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch_npu/distributed/distributed_c10d.py:108: UserWarning: HCCL doesn't support gather at the moment. Implemented with allgather instead.
warnings.warn("HCCL doesn't support gather at the moment. Implemented with allgather instead.")
[rank1]: Traceback (most recent call last):
[rank1]: File "/home/ma-user/work/nlp/x00958244/MindSpeed-MM/pretrain_transformers.py", line 119, in
[rank1]: pretrain(
[rank1]: File "/home/ma-user/work/nlp/x00958244/MindSpeed-MM/mindspeed_mm/training.py", line 254, in pretrain
[rank1]: save_checkpoint(
[rank1]: File "/home/ma-user/work/nlp/x00958244/MindSpeed-MM/megatron/training/checkpointing.py", line 467, in save_checkpoint
[rank1]: torch.distributed.checkpoint.save(
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/logger.py", line 87, in wrapper
[rank1]: result = func(*args, **kwargs)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 465, in inner_func
[rank1]: return func(*args, **kwargs)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict_saver.py", line 176, in save
[rank1]: return _save_state_dict(
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict_saver.py", line 367, in _save_state_dict
[rank1]: return distW.all_reduce("write", write_data, finish_checkpoint)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 259, in all_reduce
[rank1]: raise final_result
[rank1]: torch.distributed.checkpoint.api.CheckpointException: CheckpointException ranks:dict_keys([0, 1])
[rank1]: Traceback (most recent call last): (RANK 0)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 239, in all_reduce
[rank1]: local_data = map_fun()
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/logger.py", line 87, in wrapper
[rank1]: result = func(*args, **kwargs)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict_saver.py", line 356, in write_data
[rank1]: all_writes = storage_writer.write_data(final_local_plan, planner)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 635, in write_data
[rank1]: return self._write_data(planner, file_queue)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 664, in _write_data
[rank1]: _write_files_from_queue(
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 406, in _write_files_from_queue
[rank1]: assert tensor.is_cpu
[rank1]: AssertionError
[rank1]: Traceback (most recent call last): (RANK 1)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/utils.py", line 239, in all_reduce
[rank1]: local_data = map_fun()
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/logger.py", line 87, in wrapper
[rank1]: result = func(*args, **kwargs)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/state_dict_saver.py", line 356, in write_data
[rank1]: all_writes = storage_writer.write_data(final_local_plan, planner)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 635, in write_data
[rank1]: return self._write_data(planner, file_queue)
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 664, in _write_data
[rank1]: _write_files_from_queue(
[rank1]: File "/home/ma-user/anaconda3/envs/PyTorch-2.5.1/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py", line 406, in _write_files_from_queue
[rank1]: assert tensor.is_cpu
[rank1]: AssertionError

Special notes for this issue/备注 (Optional / 选填)