| chore: add VS Code devcontainer configuration Co-authored-by: wxy1105<wangxinyu136@huawei.com> # message auto-generated for no-merge-commit merge: !4892 merge devcontainer_0730 into master chore: add VS Code devcontainer configuration Created-by: wxy1105 Commit-by: wxy1105 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a complete VS Code Dev Container for MindSpeed-LLM, including: CANN, PyTorch, TorchNPU, Triton-Ascend, MindSpeed, Megatron-LM, and FSDPTurbo. NPU driver/device passthrough. Sequential pip mirror fallback and bounded Megatron-LM clone retries. External repositories arranged alongside MindSpeed-LLM under /workspace. Automatic dependency installation and Megatron symlink refresh. This provides a reproducible NPU development and testing environment. ## Does this PR introduce any user-facing change? Yes. Users can now open the repository with VS Code Dev Containers and work in a preconfigured NPU environment. Usage instructions, configuration, version customization, and troubleshooting are documented in: .devcontainer/README.md The existing repository is mounted at /workspace/MindSpeed-LLM; external dependencies are placed under /workspace. Dependency conflicts reported by pip check are warnings and do not block container initialization. ## How was this patch tested? The JSON configuration, Dockerfile syntax/diffs, paths, and dependency-source logic were statically validated. Dev Container builds were also used to verify image construction, dependency installation, post-create initialization, and identified network/dependency failure handling. Verify manually with: npu-smi info python3 -c "import torch, torch_npu; print(torch.__version__, torch.npu.is_available())" python3 -c "import mindspeed, megatron" readlink /workspace/MindSpeed-LLM/megatron python3 -m pip check See merge request: Ascend/MindSpeed-LLM!4892 | 11 天前 |
| docs: change pr template Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !4299 merge master_pr_template3 into master docs: change pr template Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: ## What this PR does / why we need it? change pr template ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4299 | 4 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 11 天前 |
| test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4909 merge master into master test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1704](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1704) ## What this PR does / why we need it? This PR moves the pretrain_qwen3_8b_general_cp_ring test script and its baseline JSON from the st directory to the pipeline directory to align with the pipeline test conventions. It also updates the deepseek4-flash checkpoint path and adds timestamped log output. ## Does this PR introduce any user-facing change? No user-facing change. The README test index table is updated to reflect the new path, but there is no impact on end-user functionality. ## How was this patch tested? The qwen3-8b cp_ring and deepseek4-flash scripts were executed successfully on an 8-NPU environment, and the results were verified against baselines. See merge request: Ascend/MindSpeed-LLM!4909 | 10 天前 |
| refactor(pytorch): delete ckpt-v1 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4831 merge master into master refactor(pytorch): delete ckpt-v1 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR removes the legacy Weight Conversion V1 implementation and completes the migration to Weight Conversion V2. The main changes include: * Remove the deprecated Weight Conversion V1 codebase. * Migrate the remaining scripts that still depended on V1 to Weight Conversion V2. * Update related README and documentation to reflect the new conversion workflow. * Clean up obsolete files and references associated with the V1 implementation. This change simplifies code maintenance and ensures that all weight conversion workflows are unified under the actively maintained V2 framework. --- ## Does this PR introduce any user-facing change? Yes. Weight Conversion V1 is no longer supported. Users should use Weight Conversion V2 for all checkpoint conversion workflows. The related README and documentation have been updated accordingly to guide users to the new conversion process. --- ## How was this patch tested? The following verification was performed: * Verified that all remaining weight conversion scripts have been successfully migrated to Weight Conversion V2. * Verified that the updated conversion workflow functions correctly after removing the V1 implementation. * Verified that related documentation and examples are consistent with the current implementation. * Regression tests and pipeline ST passed successfully. See merge request: Ascend/MindSpeed-LLM!4831 | 11 天前 |
| docs(pytorch): add Disclaimer of docker overview Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4935 merge master into master docs(pytorch): add Disclaimer of docker overview Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? Add a disclaimer to the Docker documentation to clarify usage scope, limitations, and user responsibilities. This is a documentation-only change and does not affect Docker images, build process, or runtime behavior. ## Does this PR introduce any user-facing change? Yes. Users will see an additional disclaimer section in the Docker documentation. There are no changes to APIs, commands, or existing usage workflows. Documentation: - <docker/OVERVIEW.md> - <docker/OVERVIEW.zh.md> ## How was this patch tested? Documentation-only change. Verified the content, formatting, and documentation rendering. No code or Docker functionality changes are introduced. See merge request: Ascend/MindSpeed-LLM!4935 | 3 天前 |
| docs(pytorch): change wechat QR code Co-authored-by: tichang<tichang@huawei.com> # message auto-generated for no-merge-commit merge: !4944 merge doc/update_llm_group_qrcode into master docs(pytorch): change wechat QR code Created-by: tichang Commit-by: tichang Merged-by: ascend-robot Description: ## What this PR does / why we need it? change wechat QR code ## Does this PR introduce any user-facing change? no introducing user-facing change ## How was this patch tested? the QR code will be effective within two weeks. See merge request: Ascend/MindSpeed-LLM!4944 | 2 天前 |
| feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 7 小时前 |
| fix(pytorch): ds4 LoRA fine-tuning gets stuck when PP is enabled Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4943 merge lora_rollback into master fix(pytorch): ds4 LoRA fine-tuning gets stuck when PP is enabled Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Revert the previous fix for the PP stuck issue during LoRA fine-tuning. The root cause is the LoRA A/B matrices are not updated when recomputation is enabled for DeepSeek V4. See PR: https://gitcode.com/Ascend/MindSpeed-LLM/pull/4921. ## Does this PR introduce any user-facing change? No. Internal bugfix, no config or interface changes. ## How was this patch tested? After reverting this modification based on the PR: https://gitcode.com/Ascend/MindSpeed-LLM/pull/4921, PP will no longer get stuck when enabled. See merge request: Ascend/MindSpeed-LLM!4943 | 4 小时前 |
| feat: Added gitleaks for sensitive information detection Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4843 merge master into master feat: Added gitleaks for sensitive information detection Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 引入gitleaks二进制离线扫描工具 2. 新增pre-commit/.gitleaks.toml配置，继承官方全部检测规则 3. 配置pre-commit钩子，提交前自动扫描密钥硬编码风险。 ## Does this PR introduce any user-facing change? 无. ## How was this patch tested? PR流水线pre-commit检测新增敏感信息检测. See merge request: Ascend/MindSpeed-LLM!4843 | 23 天前 |
| feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 7 小时前 |
| [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4448 merge master into master [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? pre-commit 工具规则配置修改：试运行阶段发现python bandit工具检测过严，超出昇腾编程规范。 此次修改重点修改bandit规则。 ## Does this PR introduce any user-facing change? 门禁codecheck检测规则精简，提升代码合入效率 ## How was this patch tested? 代码扫描工具配置，不涉及代码仓功能 See merge request: Ascend/MindSpeed-LLM!4448 | 3 个月前 |
| docs(fsdp2): fix documentation format issues and improve readability Co-authored-by: wangjiangben<wangjiangben@huawei.com> # message auto-generated for no-merge-commit merge: !4476 merge docs/fix-fsdp2-docs-format into master docs(fsdp2): fix documentation format issues and improve readability Created-by: wangjiangben Commit-by: wangjiangben Merged-by: ascend-robot Description: ## Summary Fix format errors in FSDP2 documentation and optimize document structure to improve readability and compliance with Markdown standards. **Format Fixes:** - Fix table format issues in quantization.md (missing header separators, column alignment errors) - Fix HTML entity syntax error in arguments.md ("ulysses" missing semicolon) - Remove extra blank lines in code blocks - Standardize indentation in example scripts **Structure Optimization:** - Unify list markers to standard Markdown - syntax - Optimize DTensor section hierarchy for better structure clarity - Convert reference links to proper Markdown link format - Split long paragraphs in quantization descriptions for better readability - Fix MD032 lint error (add blank line before list) **Files Changed:** - arguments.md: Fix HTML entity syntax - fsdp2_basic_features.md: Optimize structure hierarchy and list format - quantization.md: Fix table format, optimize description text See merge request: Ascend/MindSpeed-LLM!4476 | 2 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 11 天前 |
| fix: Fix the issue where LoRA A/B matrices are not updated when recomputation is enabled for DeepSeek V4. Co-authored-by: xiejiahao2333<xiejiahao11@huawei.com> # message auto-generated for no-merge-commit merge: !4921 merge fix-lora-recompute-hook into master fix: Fix the issue where LoRA A/B matrices are not updated when recomputation is enabled for DeepSeek V4. Created-by: xiejiahao2333 Commit-by: xiejiahao2333 Merged-by: ascend-robot Description: https://gitcode.com/Ascend/MindSpeed-LLM/issues/1769 ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. This PR fixes an issue where the LoRA A/B matrices are not updated when recomputation is enabled for DeepSeek V4. 一、**Issue:**：During LoRA fine-tuning with DeepSeek V4 using PP2 TP1 EP8, the LoRA B matrix on the PP0 partition remains all zeros and is not updated. 二、**Root cause:**： 1. The input of PP0 comes from the frozen embedding layer. In the LoRA scenario, the embedding layer itself does not participate in training. The hook that sets requires_grad=True for the embedding output is only registered when recompute_method == block, as shown below:  2. PP1 is not affected because the tensor received through pipeline parallel communication explicitly has requires_grad=True, as shown below:  ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. **Solution：**：This PR relaxes the registration condition of the LoRA embedding hook so that the embedding output can correctly participate in gradient computation under the affected recomputation settings.  After this fix, the LoRA B matrix on PP0 is no longer all zeros and is updated correctly during training. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. Commit ID：1ed31e0e36aefdd2e868ec8e865164fd0b030480 Training configuration: { "task_type": "sft", "model_type": "deepseek4", "peft_type": "full", "custom": { "use_mp_training": true, "train_dir": "/dsV4/train_chatml.json" }, "train": { "tokenizer-name-or-path": "/models/deepseek-ai/DeepSeek-V4-Flash-BF16", "tokenizer-type": "PretrainedFromHF", "load": "/dsV4/weight/DeepSeek-V4-Flash-0731-Megatron-pp4tp1ep4", "save": "./save_lora", "prompt_type": "deepseek4", "finetune": true, "no-load-optim": true, "no-load-rng": true, "seed": 1234, "split": "100,0,0", "lr": 1e-5, "min-lr": "1.0e-7", "lr-decay-style": "cosine", "lr-warmup-iters": 5, "bf16": true, "weight-decay": "1e-2", "clip-grad": "1.0", "adam-beta1": "0.9", "adam-beta2": "0.999", "train-iters": 2000, "micro-batch-size": 1, "global-batch-size": 128, "tensor-model-parallel-size": 1, "pipeline-model-parallel-size": 4, "expert-model-parallel-size": 4, "expert-tensor-parallel-size": 1, "compress-rope-theta": 160000.0, "context-parallel-size": 1, "rope_factor": 16.0, "sequence-parallel": true, "seq-length": 4096, "mtp-num-layers": 0, "initial-loss-scale": 65536.0, "log-interval": 1, "eval-interval": 1000, "eval-iters": 1000, "save-interval": 50, "use-flash-attn": true, "use-distributed-optimizer": true, "tensorboard-queue-size": 1000, "tensorboard-dir": "./log_dir", "recompute-granularity": "full", "swap-optimizer": true, "use-mcore-models": true, "is-instruction-dataset": true, "use-triton-mhc": false, "ckpt-format": "torch", "moe-permute-fusion": true, "moe-grouped-gemm": true, "moe-shared-expert-overlap": false, "recompute-method": "uniform", "recompute-num-layers": 1, "no-save-optim": true, "no-save-rng": true, "log-throughput": true, "fix-router": false, "lora-r": 16, "lora-alpha": 32, "lora-fusion": true, "lora-target-modules": ["linear_q", "linear_kv", "linear_fc1", "linear_fc2", "linear_q_up_proj", "linear_o_down_proj", "linear_o_up_proj"], "moe-token-dispatcher-type": "alltoall_seq" } } Loss curve:  Update status of the A/B matrices：  vLLM test result after training：  See merge request: Ascend/MindSpeed-LLM!4921 | 1 天前 |
| docs: update README and CONTRIBUTING.md Co-authored-by: xuwenyue<xuwenyue1@huawei.com> # message auto-generated for no-merge-commit merge: !4440 merge master into master docs: update README and CONTRIBUTING.md Created-by: xuwenyue Commit-by: xuwenyue Merged-by: ascend-robot Description: ## What this PR does / why we need it? update README, CONTRIBUTING.md, FAQ.md, features/README.md, performance_tuning.md ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4440 | 2 个月前 |
| update LICENSE. add license Signed-off-by: fengliangjun <fengliangjun@huawei.com> | 1 年前 |
| fix: migrate llama2 convert scripts to v2 Co-authored-by: wanggangguo<wanggangguo@huawei.com> # message auto-generated for no-merge-commit merge: !4700 merge debug5 into master fix: migrate llama2 convert scripts to v2 Created-by: isfrapples Commit-by: wanggangguo Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. Migrate gemma2, llama2, plm, qwen3 lora checkpoint scripts to v2. 2. Add phi3.5-moe support in convert_ckpt_v2.py and model_cfg.json. 3. Add missing add_qkv_bias config for phi3.5-moe in model_cfg.json. ## Does this PR introduce any user-facing change? 1. No, only internal example scripts are updated to use the new converter. 2. All existing interfaces and behaviors remain unchanged. ## How was this patch tested? 1. All migrated scripts share the same CLI pattern as existing v2 tests. 2. phi3.5-moe changes are validated by existing CI configurations. fixes#1580 See merge request: Ascend/MindSpeed-LLM!4700 | 1 个月前 |
| docs(pytorch): rename install_guide and updated the latest link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4910 merge master into master docs(pytorch): rename install_guide and updated the latest link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? rename install_guide and updated the latest link ## Does this PR introduce any user-facing change? rename install_guide and updated the latest link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4910 | 10 天前 |
| docs(pytorch): updated incorrect link Co-authored-by: LQ1206<liuqian164@h-partners.com> # message auto-generated for no-merge-commit merge: !4927 merge master into master docs(pytorch): updated incorrect link Created-by: LQ1206 Commit-by: LQ1206 Merged-by: ascend-robot Description: ## What this PR does / why we need it? updated incorrect link ## Does this PR introduce any user-facing change? updated incorrect link ## How was this patch tested? No test See merge request: Ascend/MindSpeed-LLM!4927 | 4 天前 |
| !2500 update notice Merge pull request !2500 from sunjunjie/master | 1 年前 |
| feat(pytorch): add hy3 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4781 merge master into master feat(pytorch): add hy3 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? - Add Hunyuan3 model adaptation. - Add Hunyuan3 model configuration. - Add Hunyuan3 weight conversion support. - Integrate Hunyuan3 into the existing model loading and training workflow. ## Does this PR introduce any user-facing change? Yes. This PR introduces support for the Hunyuan3 model. Users can now use MindSpeed-LLM to perform weight conversion and training for Hunyuan3 following the existing workflow. No user-facing changes are introduced for existing supported models. ## How was this patch tested? The following tests have been completed: - Pipeline ST passed. - Verified that Hunyuan3 weight conversion completes successfully. - Verified that the converted checkpoint can be loaded successfully. - Verified that the training workflow starts successfully after conversion. See merge request: Ascend/MindSpeed-LLM!4781 | 16 天前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 15 天前 |
| feature(pytorch): FSDP2 support hardware-adaptive execution Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4343 merge fsdp2_gpu into master feature(pytorch): FSDP2 support hardware-adaptive execution Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? FSDP2 support hardware-adaptive execution ## Does this PR introduce any user-facing change? No ## How was this patch tested? pipeline test passed See merge request: Ascend/MindSpeed-LLM!4343 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| feat(pytorch): add DeepSeek4 fine-tuning template Co-authored-by: HanhuiChen<chenhanhui1@h-partners.com> # message auto-generated for no-merge-commit merge: !4436 merge dsv4 into master feat(pytorch): add DeepSeek4 fine-tuning template Created-by: HANHU1CHEN Commit-by: HanhuiChen Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a fine-tuning template for the DeepSeek4 model series to support its specific prompt format, including thinking mode, tool calling (DSML format), and reasoning effort control. ## Does this PR introduce any user-facing change? Yes — users can now select --prompt-type deepseek4 to fine-tune DeepSeek4 models. Two new behaviors are also exposed: - --enable-thinking controls thinking vs chat mode - --reasoning-effort {max,high} inserts a max-effort instruction prefix; only valid when thinking is enabled - --drop-thinking controls whether reasoning content is kept in each turn ## How was this patch tested? Tested with byte-level alignment against the official encoding_dsv4 script. See merge request: Ascend/MindSpeed-LLM!4436 | 3 个月前 |
| [pytorch][feature]Add RL data processing Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3646 merge fenzhi into master [pytorch][feature]Add RL data processing Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: add preprocess prmompt 修改模板路径 See merge request: Ascend/MindSpeed-LLM!3646 | 8 个月前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 15 天前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 4 个月前 |
| feat: third-party dependency reduction Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4887 merge requirements into master feat: third-party dependency reduction Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Non-essential Dependency Cleanup. ## Does this PR introduce any user-facing change? Remove irrelevant dependencies to reduce the difficulty for customers to get started with installation. ## How was this patch tested? CI and Nightly-CI-Component-pipeline_MindSpeed-LLM_master. See merge request: Ascend/MindSpeed-LLM!4887 | 3 天前 |
| add rl ci Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3951 merge master_rl_ci into master [pytorch][feature]Adapt RL for vllm0110 and add CI Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: [pytorch][feature]Adapt RL for vllm0110 and add CI See merge request: Ascend/MindSpeed-LLM!3951 | 7 个月前 |
| [pytorch][build]update setup.py version and python support range master Co-authored-by: tichang<tichang@huawei.com> # message auto-generated for no-merge-commit merge: !4076 merge chore/update_setup.py_version_and_pyversion_info into master [pytorch][build]update setup.py version and python support range master Created-by: tichang Commit-by: tichang Merged-by: ascend-robot Description: chore(setup): update setup.py version and python support range 在master分支修改了setup.py中的版本信息和python支持版本 See merge request: Ascend/MindSpeed-LLM!4076 | 6 个月前 |
| feat: add dsv4 fsdp sh && performance optimization Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4859 merge master into master feat: add dsv4 fsdp sh && performance optimization Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? ### Background As large language model parameters continue to grow, new model architectures like DeepSeek V4 impose higher demands on distributed training. MindSpeed-LLM, as a distributed training suite for LLMs based on the Ascend NPU ecosystem, needs to continuously adapt to new models and optimize training performance. Currently, the repository does not provide a complete training script with performance optimizations for DeepSeek V4 using the FSDP (Fully Sharded Data Parallel) backend, which limits users' ability to efficiently train DeepSeek V4 on Ascend NPUs. ### Detailed Changes This PR introduces the following changes: - **Add DeepSeek V4 FSDP training script** – provides a ready‑to‑use startup script (dsv4 fsdp sh) for FSDP‑based distributed training of DeepSeek V4, reducing the effort required to adapt the model to the FSDP backend. - **Performance optimization** – performs targeted tuning for the DeepSeek V4 model architecture on the FSDP backend, including but not limited to: 1. Integrating the MHC fused operator. 2. Integrating Torch's fused AdamW optimizer. 3. Removing redundant transpose operations. 4. Integrating the SwiGLU_Limit fused operator. - **Training workflow validation** – ensures that the new script covers the basic pre‑training path, confirming correctness and stability. ## Does this PR introduce any user-facing change? Yes. This PR provides a new entry‑point script for DeepSeek V4 FSDP training. Users can directly use this script to launch distributed pre‑training tasks for DeepSeek V4 without manually writing complex distributed configurations. ## How was this patch tested? ### Functional Correctness Verification - Run the newly added dsv4 fsdp sh script on an Ascend NPU cluster to verify that DeepSeek V4 can start FSDP distributed training successfully. - Monitor training loss to ensure it converges as expected, confirming the correctness of the training logic. ### Performance Verification - Compare training throughput (tokens/s) and memory usage before and after the optimization. - Validate that the performance improvements meet the expected targets. See merge request: Ascend/MindSpeed-LLM!4859 | 11 天前 |
