Thanks for sending an issue! Please fill in the following template to help quickly solve your problem.

Describe the current behavior / 问题描述 (Mandatory / 必填)

目前使用master分支对qwen3-32b模型进行lora微调训练后，无法将megatron权重转为huggingface权重，请问现在微调脚本中不加--lora-ckpt-filter参数，是否默认保存的是全部权重，如果是，使用以下转换脚本，仍然报错
`source /usr/local/Ascend/ascend-toolkit/set_env.sh
export CUDA_DEVICE_MAX_CONNECTIONS=1

python convert_ckpt_v2.py
--load-model-type mg
--save-model-type hf
--load-dir ./ckpt/qwen3_32b/lora/1/tp8_pp1/20251219_18
--save-dir ./hf_weights/qwen3_32b_lora_1k_easy_iter140/
--model-type-hf qwen3 `
如果不是，请提供一个lora训练后megatron权重转为huggingface权重的脚本示例，现在convert_ckpt_v2.py不是不支持lora权重合并？

Environment / 环境信息 (Mandatory / 必填)

硬件：910B2
CANN：8.3.RC1
python: 3.11
torch: 2.7.1
torch_npu: 2.7.1
MindeSpeed-LLM: master分支

Steps to reproduce the issue / 重现步骤 (Mandatory / 必填)

lora微调脚本如下：
`export HCCL_CONNECT_TIMEOUT=1800
export CUDA_DEVICE_MAX_CONNECTIONS=1

NPUS_PER_NODE=8
MASTER_ADDR=localhost
MASTER_PORT=6014
NNODES=1
NODE_RANK=0
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))

CKPT_LOAD_DIR="./model_weights/qwen3_32b_lora_1k_easy_iter140_tp8_pp1"
CKPT_SAVE_DIR="ckpt/qwen3_32b/lora-id1-2/tp8_pp1/"
DATA_PATH="./finetune_dataset/sft"
TOKENIZER_PATH="/hpfs/weights/Qwen3-32B/"
LOG_PATH="./logs/qwen3_32b/lora/}"
TENSORBOARD_DIR="./tensorboard/qwen3_32b/lora/"
WANDB_DIR="./wandb/qwen3_32b/lora/"
LOG_FILE="${LOG_PATH}/${NODE_RANK}.log"

folder_list=(
$CKPT_SAVE_DIR
$LOG_PATH
$PROFILE_DIR
$TENSORBOARD_DIR
$WANDB_DIR
)

for folder_path in "${folder_list[@]}"; do if [ ! -d "$folder_path" ]; then
# 创建文件夹
mkdir -p "$folder_path" else echo "$folder_path is exists"
fi
done

TP=8
PP=1
SEQ_LENGTH=4096
TRAIN_ITERS=120
EVAL_ITERS=15
SAVE_INTERVAL=5
EVAL_INTERAL=5
MBS=4
GBS=8

DISTRIBUTED_ARGS="
--nproc_per_node $NPUS_PER_NODE
--nnodes $NNODES
--node_rank $NODE_RANK
--master_addr $MASTER_ADDR
--master_port $MASTER_PORT
"

OPTIMIZE_ARGS="
--use-flash-attn
--use-fused-rotary-pos-emb
--use-rotary-position-embeddings
--use-fused-swiglu
--use-fused-rmsnorm
--no-masked-softmax-fusion
--use-distributed-optimizer
--sequence-parallel
"

TRAIN_ARGS="
--micro-batch-size ${MBS}
--global-batch-size ${GBS}
--lr 1.25e-5
--lr-decay-style cosine
--min-lr 1.25e-7
--weight-decay 1e-1
--lr-warmup-fraction 0.01
--attention-dropout 0.0
--init-method-std 0.01
--hidden-dropout 0.0
--clip-grad 1.0
--adam-beta1 0.9
--adam-beta2 0.95
--initial-loss-scale 4096
--seed 42
--bf16
--train-iters ${TRAIN_ITERS}
--seq-length ${SEQ_LENGTH}
--no-shared-storage
"

MODEL_PARALLEL_ARGS="
--tensor-model-parallel-size ${TP}
--pipeline-model-parallel-size ${PP}
"

GPT_ARGS="
--use-mcore-models
--spec mindspeed_llm.tasks.models.spec.qwen3_spec layer_spec
--kv-channels 128
--qk-layernorm
--tokenizer-name-or-path ${TOKENIZER_PATH}
--max-position-embeddings ${SEQ_LENGTH}
--num-layers 64
--hidden-size 5120
--ffn-hidden-size 25600
--num-attention-heads 64
--tokenizer-type PretrainedFromHF
--make-vocab-size-divisible-by 1
--padded-vocab-size 151936
--rotary-base 1000000
--untie-embeddings-and-output-weights
--disable-bias-linear
--position-embedding-type rope
--normalization RMSNorm
--swiglu
--attention-softmax-in-fp32
--no-gradient-accumulation-fusion
--group-query-attention
--num-query-groups 8
"

DATA_ARGS="
--data-path $DATA_PATH
--split 90,10,0
"

OUTPUT_ARGS="
--load ${CKPT_LOAD_DIR}
--save ${CKPT_SAVE_DIR}
--log-interval 1
--save-interval ${SAVE_INTERVAL}
--eval-interval ${EVAL_INTERAL}
--eval-iters ${EVAL_ITERS}
--no-load-optim
--no-load-rng
--log-throughput
--log-params-norm
--tensorboard-log-interval 1
--log-timers-to-tensorboard
--log-memory-to-tensorboard
--tensorboard-dir ${TENSORBOARD_DIR}
--wandb-project test
--wandb-exp-name qwen3-32b-lora
--wandb-save-dir ${WANDB_DIR}
--use-wandb
"

TUNE_ARGS="
--finetune
--stage sft
--is-instruction-dataset
--tokenizer-not-use-fast
--prompt-type qwen3
--no-pad-to-seq-lengths
--lora-r 16
--lora-alpha 32
--lora-fusion
--lora-target-modules linear_qkv linear_proj linear_fc1 linear_fc2
"

torchrun $DISTRIBUTED_ARGS posttrain_gpt.py
$GPT_ARGS
$DATA_ARGS
$OUTPUT_ARGS
$OPTIMIZE_ARGS
$TRAIN_ARGS
$TUNE_ARGS
$MODEL_PARALLEL_ARGS
--distributed-backend nccl
2>&1 | tee ${LOG_FILE}`

Describe the expected behavior / 预期结果 (Mandatory / 必填)

qwen3-32b lora微调后将megatron权重转为huggingface权重

Related log / screenshot / 日志 / 截图 (Mandatory / 必填)

/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:347: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.load, torch.set_default_device, torch.get_device_module, torch.sparse_compressed_tensor, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************

warnings.warn(msg, ImportWarning)
/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:276: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
Using /root/.cache/torch_extensions/py311_cpu as PyTorch extensions root...
/usr/local/python3.11.13/lib/python3.11/site-packages/torch_npu/dynamo/torchair/init.py:8: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81.
import pkg_resources
/hpfs/lqh/MindSpeed-LLM/mindspeed_llm/core/transformer/transformer_block.py:30: DeprecationWarning: The 'megatron.core.transformer.custom_layers.transformer_engine'
module is deprecated and will be removed in 0.10.0. Please use
'megatron.core.extensions.transformer_engine' instead.
from megatron.core.transformer.custom_layers.transformer_engine import TENorm
INFO:root:Arguments: Namespace(load_model_type='mg', save_model_type='hf', load_dir='./ckpt/qwen3_32b/lora/1/tp8_pp1/20251219_18', save_dir='./hf_weights/qwen3_32b_lora_1k_easy_iter140/', model_type_hf='qwen3', target_tensor_parallel_size=1, target_pipeline_parallel_size=1, target_expert_parallel_size=1, expert_tensor_parallel_size=None, num_layers_per_virtual_pipeline_stage=None, moe_grouped_gemm=False, noop_layers=None, mtp_num_layers=0, num_layer_list=None, moe_tp_extend_ep=False, mla_mm_split=False, schedules_method=None, first_k_dense_replace=None, num_layers=None, transformer_impl='local')
Warning: The current version of the file storing weights is old, and it is relanded due to internal bug of torch and compatibility issue. We will deprecate the loading support for this type of file in the future, please use newer torch to re-store the weight file.
INFO:root:Megatron arguments is loaded from ./ckpt/qwen3_32b/lora/1/tp8_pp1/20251219_18/iter_0000140/mp_rank_05/model_optim_rng.pt

INFO:root:###### pprank->hf layer: defaultdict(None, {0: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]})
INFO:root:###### HF layer to (pp_rank, local_idx) mapping: defaultdict(None, {0: (0, 0), 1: (0, 1), 2: (0, 2), 3: (0, 3), 4: (0, 4), 5: (0, 5), 6: (0, 6), 7: (0, 7), 8: (0, 8), 9: (0, 9), 10: (0, 10), 11: (0, 11), 12: (0, 12), 13: (0, 13), 14: (0, 14), 15: (0, 15), 16: (0, 16), 17: (0, 17), 18: (0, 18), 19: (0, 19), 20: (0, 20), 21: (0, 21), 22: (0, 22), 23: (0, 23), 24: (0, 24), 25: (0, 25), 26: (0, 26), 27: (0, 27), 28: (0, 28), 29: (0, 29), 30: (0, 30), 31: (0, 31), 32: (0, 32), 33: (0, 33), 34: (0, 34), 35: (0, 35), 36: (0, 36), 37: (0, 37), 38: (0, 38), 39: (0, 39), 40: (0, 40), 41: (0, 41), 42: (0, 42), 43: (0, 43), 44: (0, 44), 45: (0, 45), 46: (0, 46), 47: (0, 47), 48: (0, 48), 49: (0, 49), 50: (0, 50), 51: (0, 51), 52: (0, 52), 53: (0, 53), 54: (0, 54), 55: (0, 55), 56: (0, 56), 57: (0, 57), 58: (0, 58), 59: (0, 59), 60: (0, 60), 61: (0, 61), 62: (0, 62), 63: (0, 63)})
INFO:root:Converting the weights of layer 0
Traceback (most recent call last):
File "/hpfs/lqh/MindSpeed-LLM/convert_ckpt_v2.py", line 75, in
main()
File "/hpfs/lqh/MindSpeed-LLM/convert_ckpt_v2.py", line 69, in main
converter.run()
File "/hpfs/lqh/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_mg2hf.py", line 1102, in run
self.read_pp_rank_weights(pp_rank, mg_weights)
File "/hpfs/lqh/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_mg2hf.py", line 1005, in read_pp_rank_weights
self.set_model_layer_attn(hf_weight_dict, mg_weights, layer, local_idx)
File "/hpfs/lqh/MindSpeed-LLM/mindspeed_llm/tasks/checkpoint/convert_mg2hf.py", line 608, in set_model_layer_attn
linear_qkv_list.append(mg_weight[(tp_rank, self.ep_rank_list[0])].pop(linear_qkv_key))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyError: 'decoder.layers.0.self_attention.linear_qkv.weight'
[ERROR] 2025-12-22-18:53:48 (PID:509285, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

Special notes for this issue/备注 (Optional / 选填)