| chore: add VS Code devcontainer configuration Co-authored-by: wxy1105<wangxinyu136@huawei.com> # message auto-generated for no-merge-commit merge: !4892 merge devcontainer_0730 into master chore: add VS Code devcontainer configuration Created-by: wxy1105 Commit-by: wxy1105 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a complete VS Code Dev Container for MindSpeed-LLM, including: CANN, PyTorch, TorchNPU, Triton-Ascend, MindSpeed, Megatron-LM, and FSDPTurbo. NPU driver/device passthrough. Sequential pip mirror fallback and bounded Megatron-LM clone retries. External repositories arranged alongside MindSpeed-LLM under /workspace. Automatic dependency installation and Megatron symlink refresh. This provides a reproducible NPU development and testing environment. ## Does this PR introduce any user-facing change? Yes. Users can now open the repository with VS Code Dev Containers and work in a preconfigured NPU environment. Usage instructions, configuration, version customization, and troubleshooting are documented in: .devcontainer/README.md The existing repository is mounted at /workspace/MindSpeed-LLM; external dependencies are placed under /workspace. Dependency conflicts reported by pip check are warnings and do not block container initialization. ## How was this patch tested? The JSON configuration, Dockerfile syntax/diffs, paths, and dependency-source logic were statically validated. Dev Container builds were also used to verify image construction, dependency installation, post-create initialization, and identified network/dependency failure handling. Verify manually with: npu-smi info python3 -c "import torch, torch_npu; print(torch.__version__, torch.npu.is_available())" python3 -c "import mindspeed, megatron" readlink /workspace/MindSpeed-LLM/megatron python3 -m pip check See merge request: Ascend/MindSpeed-LLM!4892 | 8 天前 |
| docs: change pr template Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !4299 merge master_pr_template3 into master docs: change pr template Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: ## What this PR does / why we need it? change pr template ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4299 | 4 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 8 天前 |
| test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4909 merge master into master test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1704](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1704) ## What this PR does / why we need it? This PR moves the pretrain_qwen3_8b_general_cp_ring test script and its baseline JSON from the st directory to the pipeline directory to align with the pipeline test conventions. It also updates the deepseek4-flash checkpoint path and adds timestamped log output. ## Does this PR introduce any user-facing change? No user-facing change. The README test index table is updated to reflect the new path, but there is no impact on end-user functionality. ## How was this patch tested? The qwen3-8b cp_ring and deepseek4-flash scripts were executed successfully on an 8-NPU environment, and the results were verified against baselines. See merge request: Ascend/MindSpeed-LLM!4909 | 7 天前 |
| refactor(pytorch): delete ckpt-v1 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4831 merge master into master refactor(pytorch): delete ckpt-v1 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR removes the legacy Weight Conversion V1 implementation and completes the migration to Weight Conversion V2. The main changes include: * Remove the deprecated Weight Conversion V1 codebase. * Migrate the remaining scripts that still depended on V1 to Weight Conversion V2. * Update related README and documentation to reflect the new conversion workflow. * Clean up obsolete files and references associated with the V1 implementation. This change simplifies code maintenance and ensures that all weight conversion workflows are unified under the actively maintained V2 framework. --- ## Does this PR introduce any user-facing change? Yes. Weight Conversion V1 is no longer supported. Users should use Weight Conversion V2 for all checkpoint conversion workflows. The related README and documentation have been updated accordingly to guide users to the new conversion process. --- ## How was this patch tested? The following verification was performed: * Verified that all remaining weight conversion scripts have been successfully migrated to Weight Conversion V2. * Verified that the updated conversion workflow functions correctly after removing the V1 implementation. * Verified that related documentation and examples are consistent with the current implementation. * Regression tests and pipeline ST passed successfully. See merge request: Ascend/MindSpeed-LLM!4831 | 9 天前 |
| docs(pytorch): add Disclaimer of docker overview Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4935 merge master into master docs(pytorch): add Disclaimer of docker overview Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? Add a disclaimer to the Docker documentation to clarify usage scope, limitations, and user responsibilities. This is a documentation-only change and does not affect Docker images, build process, or runtime behavior. ## Does this PR introduce any user-facing change? Yes. Users will see an additional disclaimer section in the Docker documentation. There are no changes to APIs, commands, or existing usage workflows. Documentation: - <docker/OVERVIEW.md> - <docker/OVERVIEW.zh.md> ## How was this patch tested? Documentation-only change. Verified the content, formatting, and documentation rendering. No code or Docker functionality changes are introduced. See merge request: Ascend/MindSpeed-LLM!4935 | 3 小时前 |
| docs(pytorch): Update Triton-Ascend version and compatibility information Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4941 merge master into master docs(pytorch): Update Triton-Ascend version and compatibility information Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Update Triton-Ascend version and compatibility information ## Does this PR introduce any user-facing change? NA. ## How was this patch tested? doc-ci and pipeline test pass. See merge request: Ascend/MindSpeed-LLM!4941 | 1 小时前 |
| fix(pytorch): Update DeepSeek-V4 LoRA fine-tuning script arguements Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4914 merge ds4_lora into master fix(pytorch): Update DeepSeek-V4 LoRA fine-tuning script arguements Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Modify PP/EP and delete MTP-related parameters. ## Does this PR introduce any user-facing change? vLLM inference accuracy is normal. ## How was this patch tested? Inference verification can answer questions normally. See merge request: Ascend/MindSpeed-LLM!4914 | 4 天前 |
| fix: release DSA ShareIndex holders after checkpoint backward Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !4925 merge master_share_index_holder into master fix: release DSA ShareIndex holders after checkpoint backward Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: Fixes #1701 ## What this PR does / why we need it? 1. Stops the checkpoint reconstruction closure from capturing the autograd ctx object. Megatron stores that closure in ctx.run_function; capturing ctx formed a reference cycle that retained DSA ShareIndex holders and their NPU TopK tensors until cyclic garbage collection ran. 2. Captures an immutable local tuple for argument reconstruction instead, preserving list/tuple input rebuilding and ShareIndex holder restoration during backward recomputation without changing checkpoint execution behavior. 3. Allows checkpoint contexts, holder dictionaries, and TopK tensors to be released normally after backward even when --manual-gc-interval is greater than one. ## Does this PR introduce any user-facing change? No. There is no CLI or API change. DSA ShareIndex training with activation recomputation no longer depends on frequent Python cyclic garbage collection to release checkpoint-held TopK tensors. ## How was this patch tested? 1. Ran the repository pre-commit hooks on mindspeed_llm/core/tensor_parallel/layers.py. Trailing-whitespace, end-of-file, large-file, merge-conflict, private-key, Ruff check/format, codespell, Pylint, Bandit, and typos checks passed. The configured local Gitleaks hook could not run because this checkout does not contain ./gitleaks. 2. Ran Python py_compile successfully on the changed file. 3. Ran a Python AST structural check confirming wrapped_run_func no longer references ctx. 4. Ran git diff --check successfully. 5. GitCode push hooks reported PASSED for commit f5a77bcf. 6. The user confirmed that changing --manual-gc-interval from 50 to 1 stopped the observed memory growth, which supports the cyclic-GC root cause. NPU training with this patch has not yet been rerun. See merge request: Ascend/MindSpeed-LLM!4925 | 1 天前 |
| feat: Added gitleaks for sensitive information detection Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4843 merge master into master feat: Added gitleaks for sensitive information detection Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 引入gitleaks二进制离线扫描工具 2. 新增pre-commit/.gitleaks.toml配置，继承官方全部检测规则 3. 配置pre-commit钩子，提交前自动扫描密钥硬编码风险。 ## Does this PR introduce any user-facing change? 无. ## How was this patch tested? PR流水线pre-commit检测新增敏感信息检测. See merge request: Ascend/MindSpeed-LLM!4843 | 20 天前 |
| docs(pytorch): updated incorrect link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4927 merge master into master docs(pytorch): updated incorrect link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? updated incorrect link ## Does this PR introduce any user-facing change? updated incorrect link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4927 | 1 天前 |
| [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4448 merge master into master [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? pre-commit 工具规则配置修改：试运行阶段发现python bandit工具检测过严，超出昇腾编程规范。 此次修改重点修改bandit规则。 ## Does this PR introduce any user-facing change? 门禁codecheck检测规则精简，提升代码合入效率 ## How was this patch tested? 代码扫描工具配置，不涉及代码仓功能 See merge request: Ascend/MindSpeed-LLM!4448 | 3 个月前 |
| docs(fsdp2): fix documentation format issues and improve readability Co-authored-by: wangjiangben<wangjiangben@huawei.com> # message auto-generated for no-merge-commit merge: !4476 merge docs/fix-fsdp2-docs-format into master docs(fsdp2): fix documentation format issues and improve readability Created-by: wangjiangben Commit-by: wangjiangben Merged-by: ascend-robot Description: ## Summary Fix format errors in FSDP2 documentation and optimize document structure to improve readability and compliance with Markdown standards. **Format Fixes:** - Fix table format issues in quantization.md (missing header separators, column alignment errors) - Fix HTML entity syntax error in arguments.md ("ulysses" missing semicolon) - Remove extra blank lines in code blocks - Standardize indentation in example scripts **Structure Optimization:** - Unify list markers to standard Markdown - syntax - Optimize DTensor section hierarchy for better structure clarity - Convert reference links to proper Markdown link format - Split long paragraphs in quantization descriptions for better readability - Fix MD032 lint error (add blank line before list) **Files Changed:** - arguments.md: Fix HTML entity syntax - fsdp2_basic_features.md: Optimize structure hierarchy and list format - quantization.md: Fix table format, optimize description text See merge request: Ascend/MindSpeed-LLM!4476 | 2 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 8 天前 |
| feat: Added gitleaks for sensitive information detection Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4843 merge master into master feat: Added gitleaks for sensitive information detection Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 引入gitleaks二进制离线扫描工具 2. 新增pre-commit/.gitleaks.toml配置，继承官方全部检测规则 3. 配置pre-commit钩子，提交前自动扫描密钥硬编码风险。 ## Does this PR introduce any user-facing change? 无. ## How was this patch tested? PR流水线pre-commit检测新增敏感信息检测. See merge request: Ascend/MindSpeed-LLM!4843 | 20 天前 |
| docs: update README and CONTRIBUTING.md Co-authored-by: xuwenyue<xuwenyue1@huawei.com> # message auto-generated for no-merge-commit merge: !4440 merge master into master docs: update README and CONTRIBUTING.md Created-by: xuwenyue Commit-by: xuwenyue Merged-by: ascend-robot Description: ## What this PR does / why we need it? update README, CONTRIBUTING.md, FAQ.md, features/README.md, performance_tuning.md ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4440 | 2 个月前 |
| update LICENSE. add license Signed-off-by: fengliangjun <fengliangjun@huawei.com> | 1 年前 |
| fix: migrate llama2 convert scripts to v2 Co-authored-by: wanggangguo<wanggangguo@huawei.com> # message auto-generated for no-merge-commit merge: !4700 merge debug5 into master fix: migrate llama2 convert scripts to v2 Created-by: isfrapples Commit-by: wanggangguo Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. Migrate gemma2, llama2, plm, qwen3 lora checkpoint scripts to v2. 2. Add phi3.5-moe support in convert_ckpt_v2.py and model_cfg.json. 3. Add missing add_qkv_bias config for phi3.5-moe in model_cfg.json. ## Does this PR introduce any user-facing change? 1. No, only internal example scripts are updated to use the new converter. 2. All existing interfaces and behaviors remain unchanged. ## How was this patch tested? 1. All migrated scripts share the same CLI pattern as existing v2 tests. 2. phi3.5-moe changes are validated by existing CI configurations. fixes#1580 See merge request: Ascend/MindSpeed-LLM!4700 | 1 个月前 |
| docs(pytorch): rename install_guide and updated the latest link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4910 merge master into master docs(pytorch): rename install_guide and updated the latest link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? rename install_guide and updated the latest link ## Does this PR introduce any user-facing change? rename install_guide and updated the latest link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4910 | 7 天前 |
| docs(pytorch): updated incorrect link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4927 merge master into master docs(pytorch): updated incorrect link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? updated incorrect link ## Does this PR introduce any user-facing change? updated incorrect link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4927 | 1 天前 |
| !2500 update notice Merge pull request !2500 from sunjunjie/master | 1 年前 |
| feat(pytorch): add hy3 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4781 merge master into master feat(pytorch): add hy3 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? - Add Hunyuan3 model adaptation. - Add Hunyuan3 model configuration. - Add Hunyuan3 weight conversion support. - Integrate Hunyuan3 into the existing model loading and training workflow. ## Does this PR introduce any user-facing change? Yes. This PR introduces support for the Hunyuan3 model. Users can now use MindSpeed-LLM to perform weight conversion and training for Hunyuan3 following the existing workflow. No user-facing changes are introduced for existing supported models. ## How was this patch tested? The following tests have been completed: - Pipeline ST passed. - Verified that Hunyuan3 weight conversion completes successfully. - Verified that the converted checkpoint can be loaded successfully. - Verified that the training workflow starts successfully after conversion. See merge request: Ascend/MindSpeed-LLM!4781 | 13 天前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 12 天前 |
| feature(pytorch): FSDP2 support hardware-adaptive execution Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4343 merge fsdp2_gpu into master feature(pytorch): FSDP2 support hardware-adaptive execution Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? FSDP2 support hardware-adaptive execution ## Does this PR introduce any user-facing change? No ## How was this patch tested? pipeline test passed See merge request: Ascend/MindSpeed-LLM!4343 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| feat(pytorch): add DeepSeek4 fine-tuning template Co-authored-by: HanhuiChen<chenhanhui1@h-partners.com> # message auto-generated for no-merge-commit merge: !4436 merge dsv4 into master feat(pytorch): add DeepSeek4 fine-tuning template Created-by: HANHU1CHEN Commit-by: HanhuiChen Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a fine-tuning template for the DeepSeek4 model series to support its specific prompt format, including thinking mode, tool calling (DSML format), and reasoning effort control. ## Does this PR introduce any user-facing change? Yes — users can now select --prompt-type deepseek4 to fine-tune DeepSeek4 models. Two new behaviors are also exposed: - --enable-thinking controls thinking vs chat mode - --reasoning-effort {max,high} inserts a max-effort instruction prefix; only valid when thinking is enabled - --drop-thinking controls whether reasoning content is kept in each turn ## How was this patch tested? Tested with byte-level alignment against the official encoding_dsv4 script. See merge request: Ascend/MindSpeed-LLM!4436 | 3 个月前 |
| [pytorch][feature]Add RL data processing Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3646 merge fenzhi into master [pytorch][feature]Add RL data processing Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: add preprocess prmompt 修改模板路径 See merge request: Ascend/MindSpeed-LLM!3646 | 8 个月前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 12 天前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| feat: third-party dependency reduction Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4887 merge requirements into master feat: third-party dependency reduction Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Non-essential Dependency Cleanup. ## Does this PR introduce any user-facing change? Remove irrelevant dependencies to reduce the difficulty for customers to get started with installation. ## How was this patch tested? CI and Nightly-CI-Component-pipeline_MindSpeed-LLM_master. See merge request: Ascend/MindSpeed-LLM!4887 | 3 小时前 |
| add rl ci Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3951 merge master_rl_ci into master [pytorch][feature]Adapt RL for vllm0110 and add CI Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: [pytorch][feature]Adapt RL for vllm0110 and add CI See merge request: Ascend/MindSpeed-LLM!3951 | 7 个月前 |
| [pytorch][build]update setup.py version and python support range master Co-authored-by: tichang<tichang@huawei.com> # message auto-generated for no-merge-commit merge: !4076 merge chore/update_setup.py_version_and_pyversion_info into master [pytorch][build]update setup.py version and python support range master Created-by: tichang Commit-by: tichang Merged-by: ascend-robot Description: chore(setup): update setup.py version and python support range 在master分支修改了setup.py中的版本信息和python支持版本 See merge request: Ascend/MindSpeed-LLM!4076 | 6 个月前 |
| feat: add dsv4 fsdp sh && performance optimization Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4859 merge master into master feat: add dsv4 fsdp sh && performance optimization Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? ### Background As large language model parameters continue to grow, new model architectures like DeepSeek V4 impose higher demands on distributed training. MindSpeed-LLM, as a distributed training suite for LLMs based on the Ascend NPU ecosystem, needs to continuously adapt to new models and optimize training performance. Currently, the repository does not provide a complete training script with performance optimizations for DeepSeek V4 using the FSDP (Fully Sharded Data Parallel) backend, which limits users' ability to efficiently train DeepSeek V4 on Ascend NPUs. ### Detailed Changes This PR introduces the following changes: - **Add DeepSeek V4 FSDP training script** – provides a ready‑to‑use startup script (dsv4 fsdp sh) for FSDP‑based distributed training of DeepSeek V4, reducing the effort required to adapt the model to the FSDP backend. - **Performance optimization** – performs targeted tuning for the DeepSeek V4 model architecture on the FSDP backend, including but not limited to: 1. Integrating the MHC fused operator. 2. Integrating Torch's fused AdamW optimizer. 3. Removing redundant transpose operations. 4. Integrating the SwiGLU_Limit fused operator. - **Training workflow validation** – ensures that the new script covers the basic pre‑training path, confirming correctness and stability. ## Does this PR introduce any user-facing change? Yes. This PR provides a new entry‑point script for DeepSeek V4 FSDP training. Users can directly use this script to launch distributed pre‑training tasks for DeepSeek V4 without manually writing complex distributed configurations. ## How was this patch tested? ### Functional Correctness Verification - Run the newly added dsv4 fsdp sh script on an Ascend NPU cluster to verify that DeepSeek V4 can start FSDP distributed training successfully. - Monitor training loss to ensure it converges as expected, confirming the correctness of the training logic. ### Performance Verification - Compare training throughput (tokens/s) and memory usage before and after the optimization. - Validate that the performance improvements meet the expected targets. See merge request: Ascend/MindSpeed-LLM!4859 | 9 天前 |
