ascend-robot[mindspore][sh]add qwen3 lora sh

acc6e22b创建于 2025年12月15日历史提交

文件	最后提交记录	最后更新时间
README.md	[pytorch][doc] gitcode url & readme update Co-authored-by: peihan_liu666<liupeihan@huawei.com>	10 个月前
ckpt_convert_qwen3_hf2mcore.sh	!3147 [mindspore][feature][master]add scripts for Qwen3 model Merge pull request !3147 from minghu111/master0815	11 个月前
ckpt_convert_qwen3_mcore2hf.sh	!3147 [mindspore][feature][master]add scripts for Qwen3 model Merge pull request !3147 from minghu111/master0815	11 个月前
data_convert_qwen3_instruction.sh	[pytorch][bugfix]in variable_seq_lengths mode, set --log-throughput to false Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com>	10 个月前
data_convert_qwen3_pretrain.sh	!3147 [mindspore][feature][master]add scripts for Qwen3 model Merge pull request !3147 from minghu111/master0815	11 个月前
pretrain_qwen3_0point6b_4K_ms.sh	!3115 【mindspore】【feature】add scripts for Qwen3 model Merge pull request !3115 from minghu111/master0807	1 年前
pretrain_qwen3_14b_4K_ms.sh	!2927 [mindspore][sh] add mindspore shell of qwen3 14b and 32b pretrain Merge pull request !2927 from chengpeng24/qwen3_master_dev	1 年前
pretrain_qwen3_14b_4K_ms_A3.sh	!3193 [mindspore][feature]add qwen3 pretrain shell scripts Merge pull request !3193 from mamba_ni/master	11 个月前
pretrain_qwen3_1point7b_4K_ms.sh	!3193 [mindspore][feature]add qwen3 pretrain shell scripts Merge pull request !3193 from mamba_ni/master	11 个月前
pretrain_qwen3_32b_4K_ms.sh	!2927 [mindspore][sh] add mindspore shell of qwen3 14b and 32b pretrain Merge pull request !2927 from chengpeng24/qwen3_master_dev	1 年前
pretrain_qwen3_32b_4K_ms_A3.sh	[pytorch][feature] Replace reset_position_ids with reset_attention_mask, and enable ring attention support when reset_attention_mask is active. Co-authored-by: mhh001<mahonghao1@huawei.com> # message auto-generated for no-merge-commit merge: !3506 merge master into master [pytorch][feature] Replace reset_position_ids with reset_attention_mask, and enable ring attention support when reset_attention_mask is active. Created-by: mhh111 Commit-by: mhh001 Merged-by: ascend-robot Description: pack/neat-pack场景ring cp支持 reset-attention-mask 对齐megatron 定长场景的支持 See merge request: Ascend/MindSpeed-LLM!3506	9 个月前
pretrain_qwen3_4b_4K_ms.sh	!3193 [mindspore][feature]add qwen3 pretrain shell scripts Merge pull request !3193 from mamba_ni/master	11 个月前
pretrain_qwen3_8b_4K_ms.sh	!3193 [mindspore][feature]add qwen3 pretrain shell scripts Merge pull request !3193 from mamba_ni/master	11 个月前
tune_qwen3_0point6b_4K_full_ms.sh	[mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Co-authored-by: wanglijun55<wanglijun54@huawei.com> # message auto-generated for no-merge-commit merge: !3666 merge branch-fixstyle into master [mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Created-by: wanglijun55 Commit-by: wanglijun55 Merged-by: ascend-robot Description: 1. qwen3微调脚本遗漏--master_addr和--node_rank两个参数，关联问题单https://e.gitee.com/mind_spore/issues/table?issue=ID3162，因而在此补充 2. 不涉及精度、性能、显存变化。 See merge request: Ascend/MindSpeed-LLM!3666	8 个月前
tune_qwen3_0point6b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前
tune_qwen3_14b_4K_full_ms.sh	[mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Co-authored-by: wanglijun55<wanglijun54@huawei.com> # message auto-generated for no-merge-commit merge: !3666 merge branch-fixstyle into master [mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Created-by: wanglijun55 Commit-by: wanglijun55 Merged-by: ascend-robot Description: 1. qwen3微调脚本遗漏--master_addr和--node_rank两个参数，关联问题单https://e.gitee.com/mind_spore/issues/table?issue=ID3162，因而在此补充 2. 不涉及精度、性能、显存变化。 See merge request: Ascend/MindSpeed-LLM!3666	8 个月前
tune_qwen3_14b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前
tune_qwen3_1point7b_4K_full_ms.sh	!3194 [mindspore][feature]add qwen3 tune shell scripts Merge pull request !3194 from mamba_ni/tune_qwen3	11 个月前
tune_qwen3_1point7b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前
tune_qwen3_32b_4K_full_ms.sh	[mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Co-authored-by: wanglijun55<wanglijun54@huawei.com> # message auto-generated for no-merge-commit merge: !3666 merge branch-fixstyle into master [mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Created-by: wanglijun55 Commit-by: wanglijun55 Merged-by: ascend-robot Description: 1. qwen3微调脚本遗漏--master_addr和--node_rank两个参数，关联问题单https://e.gitee.com/mind_spore/issues/table?issue=ID3162，因而在此补充 2. 不涉及精度、性能、显存变化。 See merge request: Ascend/MindSpeed-LLM!3666	8 个月前
tune_qwen3_32b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前
tune_qwen3_4b_4K_full_ms.sh	[mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Co-authored-by: wanglijun55<wanglijun54@huawei.com> # message auto-generated for no-merge-commit merge: !3666 merge branch-fixstyle into master [mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Created-by: wanglijun55 Commit-by: wanglijun55 Merged-by: ascend-robot Description: 1. qwen3微调脚本遗漏--master_addr和--node_rank两个参数，关联问题单https://e.gitee.com/mind_spore/issues/table?issue=ID3162，因而在此补充 2. 不涉及精度、性能、显存变化。 See merge request: Ascend/MindSpeed-LLM!3666	8 个月前
tune_qwen3_4b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前
tune_qwen3_8b_4K_full_ms.sh	[mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Co-authored-by: wanglijun55<wanglijun54@huawei.com> # message auto-generated for no-merge-commit merge: !3666 merge branch-fixstyle into master [mindspore][sh] add --master_addr and --node_rank for qwen3 tune sh Created-by: wanglijun55 Commit-by: wanglijun55 Merged-by: ascend-robot Description: 1. qwen3微调脚本遗漏--master_addr和--node_rank两个参数，关联问题单https://e.gitee.com/mind_spore/issues/table?issue=ID3162，因而在此补充 2. 不涉及精度、性能、显存变化。 See merge request: Ascend/MindSpeed-LLM!3666	8 个月前
tune_qwen3_8b_4K_lora_ms.sh	[mindspore][sh]add qwen3 lora sh Co-authored-by: nomiz<1606135114@qq.com> # message auto-generated for no-merge-commit merge: !3916 merge 2.3.0 into 2.3.0 [mindspore][sh]add qwen3 lora sh Created-by: nomiz Commit-by: nomiz Merged-by: ascend-robot Description: [mindspore][sh]add qwen3 lora sh 验证 tune_qwen3_0.6b_lora 精度100步对齐 ![屏幕截图 2025-12-15 192324.jpg](https://raw.gitcode.com/user-images/assets/7623105/2b3e3437-bb71-47c2-ba2b-4a4a29674d32/屏幕截图_2025-12-15_192324.jpg '屏幕截图 2025-12-15 192324.jpg') See merge request: Ascend/MindSpeed-LLM!3916	7 个月前

1 环境配置

MindSpeed-LLM MindSpore后端的安装步骤参考：基础安装指导

2 权重转换

2.1 权重下载

从huggingface(以Qwen3-0.6B为例)下载模型权重和其它配置文件，若需要在开源权重上继续预训练、微调、推理，也请下载网络模型文件。

2.2 权重转换

提供脚本将huggingface开源权重转换为mcore权重，用于训练、推理、评估等任务。使用方法如下，请根据实际需要的TP/PP等切分策略和权重路径修改权重转换脚本：

cd MindSpeed-LLM
bash examples/mindspore/qwen3/ckpt_convert_qwen3_hf2mcore.sh

运行脚本后，预期会看到类似以下的日志输出，表示权重转换成功：

successfully saved checkpoint from iteration 1 to ./model_weights/qwen3_mcore/
INFO:root:Done!

注：MindSpore 后端转换出的模型权重无法用于 Torch后端训练或推理。

3 数据预处理

当前MindSpore后端已完全支持MindSpeed-LLM的多种任务场景下的数据预处理，数据预处理指南参见数据预处理。

3.1 预训练数据处理

（以alpaca数据集为例）修改data_convert_qwen3_pretrain.sh预训练脚本

配置好数据输入/输出路径、tokenizer模型路径即可启动：

bash examples/mindspore/qwen3/data_convert_qwen3_pretrain.sh

预训练数据集处理结果如下：

./dataset/alpaca_text_document.bin
./dataset/alpaca_text_document.idx

3.2 微调数据处理

（以alpaca数据集为例）修改data_convert_qwen3_instruction.sh微调脚本

配置好数据输入/输出路径、tokenizer模型路径即可启动：

bash examples/mindspore/qwen3/data_convert_qwen3_instruction.sh

微调数据集处理结果如下：

./finetune_dataset/alpaca_packed_attention_mask_document.bin
./finetune_dataset/alpaca_packed_attention_mask_document.idx
./finetune_dataset/alpaca_packed_input_ids_document.bin
./finetune_dataset/alpaca_packed_input_ids_document.idx
./finetune_dataset/alpaca_packed_labels_document.bin
./finetune_dataset/alpaca_packed_labels_document.idx

4 训练

4.1 预训练

在pretrain_qwen3_0point6b_4K_ms.sh脚本中修改相关参数，并执行脚本

cd MindSpeed-LLM
bash examples/mindspore/qwen3/pretrain_qwen3_0point6b_4K_ms.sh

变量名	含义
MASTER_ADDR	多机情况下主节点IP
NODE_RANK	多机下，各机对应节点序号
DATA_PATH	预处理后的数据路径
TOKENIZER_PATH	Qwen3 0.6b tokenizer目录
CKPT_LOAD_DIR	初始权重加载，如无初始权重则随机初始化
CKPT_SAVE_DIR	训练中权重保存位置
TRAIN_ITERS	训练迭代步数

注：0.6b模型规模较小，一般单机即可

4.2 微调

在tune_qwen3_0point6b_4K_full_ms.sh脚本中修改相关参数，并执行脚本

cd MindSpeed-LLM
bash examples/mindspore/qwen3/tune_qwen3_0point6b_4K_full_ms.sh

变量名	含义
MASTER_ADDR	多机情况下主节点IP
NODE_RANK	多机下，各机对应节点序号
DATA_PATH	预处理后的数据路径
TOKENIZER_PATH	Qwen3 0.6b tokenizer目录
CKPT_LOAD_DIR	初始权重加载，如无初始权重则随机初始化
CKPT_SAVE_DIR	训练后的权重保存位置
TRAIN_ITERS	训练迭代步数

注1：0.6b模型规模较小，一般单机即可
注2：CKPT_LOAD_DIR选择加载预训练保存后的权重