| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
docs(pytorch): rename install_guide and updated the latest link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4910 merge master into master docs(pytorch): rename install_guide and updated the latest link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? rename install_guide and updated the latest link ## Does this PR introduce any user-facing change? rename install_guide and updated the latest link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4910 | 8 天前 | |
fix(pytorch): fix deepseekv4 script tp Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4431 merge master into master fix(pytorch): fix deepseekv4 script tp Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Update the dsv4 script: change tp parameter from 2 to 1 due to a bug in the weight conversion where the etp parameter does not take effect. ## Does this PR introduce any user-facing change? change tp parameter from 2 to 1 ## How was this patch tested? The new parameters have been verified at the customer site. See merge request: Ascend/MindSpeed-LLM!4431 | 3 个月前 | |
feat(pytorch): add dsv4 mg2hf Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4458 merge master into master feat(pytorch): add dsv4 mg2hf Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. add dsv4 mg2hf - only support pp - only support etp = 1 or tp = 1 2. fix dsv4 hf2mg vpp ## Does this PR introduce any user-facing change? if use base model of dsv4 to do mg2hf convert, please set --model-type-hf with deepseek4_base ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-LLM!4458 | 2 个月前 | |
feat(pytorch): add deepseekv4 fp8 to bf16 dequant tools Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4422 merge master into master feat(pytorch): add deepseekv4 fp8 to bf16 dequant tools Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-LLM!4422 | 3 个月前 | |
feat(pytorch): add DeepSeek4 fine-tuning template Co-authored-by: HanhuiChen<chenhanhui1@h-partners.com> # message auto-generated for no-merge-commit merge: !4436 merge dsv4 into master feat(pytorch): add DeepSeek4 fine-tuning template Created-by: HANHU1CHEN Commit-by: HanhuiChen Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a fine-tuning template for the DeepSeek4 model series to support its specific prompt format, including thinking mode, tool calling (DSML format), and reasoning effort control. ## Does this PR introduce any user-facing change? Yes — users can now select --prompt-type deepseek4 to fine-tune DeepSeek4 models. Two new behaviors are also exposed: - --enable-thinking controls thinking vs chat mode - --reasoning-effort {max,high} inserts a max-effort instruction prefix; only valid when thinking is enabled - --drop-thinking controls whether reasoning content is kept in each turn ## How was this patch tested? Tested with byte-level alignment against the official encoding_dsv4 script. See merge request: Ascend/MindSpeed-LLM!4436 | 3 个月前 | |
refactor(pytorch): update deepseek4 shell Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4423 merge master into master refactor(pytorch): update deepseek4 shell Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-LLM!4423 | 3 个月前 | |
refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 13 天前 | |
fix(pytorch): Update DeepSeek-V4 LoRA fine-tuning script arguements Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4914 merge ds4_lora into master fix(pytorch): Update DeepSeek-V4 LoRA fine-tuning script arguements Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Modify PP/EP and delete MTP-related parameters. ## Does this PR introduce any user-facing change? vLLM inference accuracy is normal. ## How was this patch tested? Inference verification can answer questions normally. See merge request: Ascend/MindSpeed-LLM!4914 | 5 天前 | |
refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 13 天前 | |
refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 13 天前 | |
refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 13 天前 |
DeepSeek-V4模型训练
2026年4月24日，DeepSeekV4系列模型DeepSeekV4-Flash和DeepSeekV4-Pro正式发布并开源，其以先进的 MoE 架构在参数效率与推理性能上实现了重要突破。MindSpeed LLM 目前已实现了 DeepSeekV4-Flash模型 的定长数据场景下的预训练支持，并同步开放源代码。目前提供预训练的实践参考，帮助用户快速上手。
注：当前实现为preview版本，部分场景存在限制，后续我们将持续完善并同步跟进 DeepSeek-V4 技术报告中的演进方向：
| 场景 | 特性 | 支持情况 | 备注 |
|---|---|---|---|
| 训练场景 | 预训练/续训 | OK | |
| 全参微调 | DOING | ||
| Lora微调 | DOING | ||
| 数据场景 | 定长数据 | OK | |
| pack数据 | DOING | ||
| 变长数据 | DOING | ||
| 切分策略 | TP | OK | 支持TP=2 |
| PP | OK | ||
| EP | OK | ||
| CP | DOING | ||
| 功能特性 | Muon优化器 | DOING |
安装指导
请参考仓库 MindSpeed LLM软件安装文档配置环境和拉取仓库代码
权重转换
-
权重下载
从 huggingface 下载权重和配置文件
-
权重转换
开源DeepSeekV4-Flash权重为FP8 mixed数据格式，训练前需要对原始权重做反量化后获得bf16格式的权重，反量化方法请参考下述脚本
cd MindSpeed-LLM bash examples/mcore/deepseek4_flash/ckpt_dequant_deepseek4_fp8_to_bf16.shMindSpeed LLM提供脚本将已经huggingface开源权重转换为mcore权重，用于训练、推理、评估等任务。 使用方法如下，请根据实际需要的TP/PP等切分策略和权重路径修改权重转换脚本
cd MindSpeed-LLM bash examples/mcore/deepseek4_flash/ckpt_convert_deepseek4_flash_hf2mcore.sh # 转换时，请指定路径到反量化后的权重
数据预处理
MindSpeed LLM提供脚本 进行数据集处理
使用方法如下，请根据实际需要修改以下参数
cd MindSpeed-LLM
bash examples/mcore/deepseek4_flash/data_convert_deepseek4_pretrain.sh
| 参数名 | 含义 |
|---|---|
| --input | 数据集路径 |
| --tokenizer-name-or-path | 模型tokenizer目录 |
| --output-prefix | 数据集处理完的输出路径及前缀名 |
预训练
MindSpeed LLM提供脚本 进行模型训练
cd MindSpeed-LLM
bash examples/mcore/deepseek4_flash/pretrain_deepseek4_flash_4k_A3_ptd.sh
用户需要根据实际情况修改脚本中以下变量
| 变量名 | 含义 |
|---|---|
| MASTER_ADDR | 多机情况下主节点IP |
| NODE_RANK | 多机下，各机对应节点序号 |
| CKPT_SAVE_DIR | 训练中权重保存路径 |
| DATA_PATH | 数据预处理后的数据路径 |
| TOKENIZER_PATH | tokenizer目录 |
| CKPT_LOAD_DIR | 权重转换保存的权重路径，为初始加载的权重，如无初始权重则随机初始化 |
全参微调
MindSpeed LLM提供脚本 进行模型训练
cd MindSpeed-LLM
bash examples/mcore/deepseek4_flash/tune_deepseek4_flash_4k_A3_ptd.sh
用户需要根据实际情况修改脚本中以下变量
| 变量名 | 含义 |
|---|---|
| MASTER_ADDR | 多机情况下主节点IP |
| NODE_RANK | 多机下，各机对应节点序号 |
| CKPT_SAVE_DIR | 训练中权重保存路径 |
| DATA_PATH | 数据预处理后的数据路径 |
| TOKENIZER_PATH | tokenizer目录 |
| CKPT_LOAD_DIR | 权重转换保存的权重路径，为初始加载的权重 |
LoRA微调
MindSpeed LLM提供脚本 进行模型训练
cd MindSpeed-LLM
bash examples/mcore/deepseek4_flash/lora_finetune_deepseek4_flash_4k_A3_ptd.sh
用户需要根据实际情况修改脚本中以下变量
| 变量名 | 含义 |
|---|---|
| MASTER_ADDR | 多机情况下主节点IP |
| NODE_RANK | 多机下，各机对应节点序号 |
| CKPT_SAVE_DIR | 训练中权重保存路径 |
| DATA_PATH | 数据预处理后的数据路径 |
| TOKENIZER_PATH | tokenizer目录 |
| CKPT_LOAD_DIR | 权重转换保存的权重路径，为初始加载的权重 |
注：LoRA微调使用指令数据集，需先执行 examples/mcore/deepseek4_flash/data_convert_deepseek4_instruction.sh 完成数据预处理，再将产物前缀填入 DATA_PATH。