| chore: add VS Code devcontainer configuration Co-authored-by: wxy1105<wangxinyu136@huawei.com> # message auto-generated for no-merge-commit merge: !4892 merge devcontainer_0730 into master chore: add VS Code devcontainer configuration Created-by: wxy1105 Commit-by: wxy1105 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Adds a complete VS Code Dev Container for MindSpeed-LLM, including: CANN, PyTorch, TorchNPU, Triton-Ascend, MindSpeed, Megatron-LM, and FSDPTurbo. NPU driver/device passthrough. Sequential pip mirror fallback and bounded Megatron-LM clone retries. External repositories arranged alongside MindSpeed-LLM under /workspace. Automatic dependency installation and Megatron symlink refresh. This provides a reproducible NPU development and testing environment. ## Does this PR introduce any user-facing change? Yes. Users can now open the repository with VS Code Dev Containers and work in a preconfigured NPU environment. Usage instructions, configuration, version customization, and troubleshooting are documented in: .devcontainer/README.md The existing repository is mounted at /workspace/MindSpeed-LLM; external dependencies are placed under /workspace. Dependency conflicts reported by pip check are warnings and do not block container initialization. ## How was this patch tested? The JSON configuration, Dockerfile syntax/diffs, paths, and dependency-source logic were statically validated. Dev Container builds were also used to verify image construction, dependency installation, post-create initialization, and identified network/dependency failure handling. Verify manually with: npu-smi info python3 -c "import torch, torch_npu; print(torch.__version__, torch.npu.is_available())" python3 -c "import mindspeed, megatron" readlink /workspace/MindSpeed-LLM/megatron python3 -m pip check See merge request: Ascend/MindSpeed-LLM!4892 | 1 个月前 |
| docs: change pr template Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !4299 merge master_pr_template3 into master docs: change pr template Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: ## What this PR does / why we need it? change pr template ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4299 | 6 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 1 个月前 |
| feat(pytorch): Upgrading Default Dependencies Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4996 merge master into master feat(pytorch): Upgrading Default Dependencies Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Upgrade Python (3.10→3.12) and PyTorch (2.7.1→2.10.0) to keep the stack current, improve performance, and apply security patches. Adjusted dependency constraints accordingly. ## Does this PR introduce any user-facing change? Yes. Users must now use **Python 3.12** and **PyTorch 2.10.0** (or compatible versions) for the development/runtime environment. Installation commands may need to be updated accordingly (e.g., conda create -n new_env python=3.12). No change to the training/inference API or command‑line arguments. ## How was this patch tested? pipeline test pass See merge request: Ascend/MindSpeed-LLM!4996 | 25 天前 |
| refactor(pytorch): delete ckpt-v1 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4831 merge master into master refactor(pytorch): delete ckpt-v1 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR removes the legacy Weight Conversion V1 implementation and completes the migration to Weight Conversion V2. The main changes include: * Remove the deprecated Weight Conversion V1 codebase. * Migrate the remaining scripts that still depended on V1 to Weight Conversion V2. * Update related README and documentation to reflect the new conversion workflow. * Clean up obsolete files and references associated with the V1 implementation. This change simplifies code maintenance and ensures that all weight conversion workflows are unified under the actively maintained V2 framework. --- ## Does this PR introduce any user-facing change? Yes. Weight Conversion V1 is no longer supported. Users should use Weight Conversion V2 for all checkpoint conversion workflows. The related README and documentation have been updated accordingly to guide users to the new conversion process. --- ## How was this patch tested? The following verification was performed: * Verified that all remaining weight conversion scripts have been successfully migrated to Weight Conversion V2. * Verified that the updated conversion workflow functions correctly after removing the V1 implementation. * Verified that related documentation and examples are consistent with the current implementation. * Regression tests and pipeline ST passed successfully. See merge request: Ascend/MindSpeed-LLM!4831 | 1 个月前 |
| feat: Update Docker Images for MindSpeed-LLM 26.1.0 Co-authored-by: wxy1105<wangxinyu136@huawei.com> # message auto-generated for no-merge-commit merge: !4946 merge docker_rp2 into master feat: Update Docker Images for MindSpeed-LLM 26.1.0 Created-by: wxy1105 Commit-by: wxy1105 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Updates the Docker stack and documentation for MindSpeed-LLM 26.1.0, including CANN 9.1.0, Python 3.12, torch-npu 2.7.1.post8, Ascend 950 support, package source optimization, and historical image tags. ## Does this PR introduce any user-facing change? Yes. Docker builds now use the latest stack by default, support 910b, a3, and 950, and include updated documentation in docker/OVERVIEW.md, docker/OVERVIEW.zh.md, and docker/supported_tags.md. ## How was this patch tested? The image verification script was executed to confirm that the image could be built successfully. See merge request: Ascend/MindSpeed-LLM!4946 | 26 天前 |
| docs: update resource links and beta terminology Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !5022 merge master_link_0907 into master docs: update resource links and beta terminology Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: Fixes #1831 Fixes #1839 ## What this PR does / why we need it? 1. Replaces the unavailable Berkeley MMLU data.tar references in the Chinese and English READMEs and model-evaluation guides with the CAIS-hosted Hugging Face download. 2. Replaces the unavailable Megatron-LM SC'21 paper references in the Chinese and English virtual-pipeline-parallel guides with arXiv 2104.04473. 3. Updates README.md feature maturity labels and their explanatory note from Prototype to beta, while preserving the surrounding Markdown structure. ## Does this PR introduce any user-facing change? Yes. Documentation readers now receive working MMLU dataset and Megatron-LM paper links, and the Chinese README displays the updated beta maturity terminology. No runtime APIs, commands, defaults, or model behavior change. ## How was this patch tested? 1. git diff --check fffc206f8351d8baa40a39134598dac94efe6767..HEAD passed. 2. Changed-file pre-commit passed trailing-whitespace, end-of-file, large-file, merge-conflict, private-key, codespell, and typos checks. Python/C++-specific hooks were not applicable. The local gitleaks-offline-scan hook could not run because the repository-required ./gitleaks executable is absent. 3. HTTP HEAD validation returned 200 for both updated resource destinations. 4. GitCode remote Git Hooks Checking passed during the force-with-lease push. See merge request: Ascend/MindSpeed-LLM!5022 | 15 天前 |
| feat(fsdp2): improve Qwen3-14B A3 pretraining support Co-authored-by: hahahascend<serverlessllm@163.com> # message auto-generated for no-merge-commit merge: !4982 merge codex/issue1354-qwen3-14b-fsdp2 into master feat(fsdp2): improve Qwen3-14B A3 pretraining support Created-by: hahahascend Commit-by: hahahascend Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optimized FSDP2 pretraining configuration for Qwen3-14B on 16 Atlas A3 NPUs with a 4K sequence length. The configuration includes: - BF16 gradient reduction to reduce communication volume. - Attention-only activation recomputation to balance memory usage and training throughput. - Parameter resharding after the forward pass to reduce peak memory usage. - ChunkLoss to reduce the peak memory consumed by vocabulary logits. - Fused RMSNorm, fused RoPE, FlashAttention, and MFU logging. - A 5,000-step training schedule with checkpoint retention settings. ## Does this PR introduce any user-facing change? Yes. It adds the following training example: - examples/fsdp2/qwen3/pretrain_qwen3_14b_4k_fsdp2_A3_optimized.sh - examples/fsdp2/qwen3/pretrain_qwen3_14b_4k_fsdp2_A3_optimized.yaml Run the example with: bash MODEL_PATH=/path/to/Qwen3-14B-Base \ DATA_PATH=/path/to/dataset_text_document \ OUTPUT_DIR=/path/to/output \ bash examples/fsdp2/qwen3/pretrain_qwen3_14b_4k_fsdp2_A3_optimized.sh The default setup uses 16 NPUs, a per-device batch size of 1, a global batch size of 16, and a sequence length of 4,096. ## How was this patch tested? The configuration completed 5,000 consecutive training steps on 16 Atlas A3 NPUs. | Metric | Result | | --- | --- | | Average loss, steps 3-102 | 0.00578247 | | Average loss, steps 4901-5000 | 0.00371331 | | Loss decrease between the two windows | 35.783% | | Loss at step 5000 | 0.003711624 | | Average MFU, steps 3-5000 | 38.199% | | Average MFU, last 100 steps | 38.213% | | Average step time, last 100 steps | 2.989 seconds | All 10 distributed checkpoint saves completed successfully, including the final global_step_5000 checkpoint. Shell syntax validation, YAML parsing, and git diff --check also passed. The results meet the acceptance criteria of decreasing loss over 5,000 steps and MFU of at least 24%. See merge request: Ascend/MindSpeed-LLM!4982 | 13 天前 |
| fix(fsdp2): restore foreach AdamW for non-EP training Co-authored-by: yanzhixiao23<yanzhixiao@h-partners.com> # message auto-generated for no-merge-commit merge: !5029 merge fix-qwen3-8b-fsdp into master fix(fsdp2): restore foreach AdamW for non-EP training Created-by: yanzhixiao23 Commit-by: yanzhixiao23 Merged-by: ascend-robot Description: ## What does this PR do / why do we need it? Fix the FSDP2 performance regression introduced by disabling AdamW foreach on NPU. The Qwen3-8B non-EP workload uses regular adamw. Disabling foreach causes parameter updates to run individually and increases optimizer step time. ## What changed? - Restore foreach=True for regular AdamW in non-EP training. - Keep foreach=False for NPU EP + FSDP2 MultiOptimizer. - Keep fused_adamw using FunctionalFusedAdamW. - Preserve the existing behavior of DeepSeek-V4 and other EP workloads. ## How was this patch tested? - Qwen3-8B FSDP2 performance recovered. - Python syntax check passed. - git diff --check passed. See merge request: Ascend/MindSpeed-LLM!5029 | 13 天前 |
| feat: Added gitleaks for sensitive information detection Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4843 merge master into master feat: Added gitleaks for sensitive information detection Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. 引入gitleaks二进制离线扫描工具 2. 新增pre-commit/.gitleaks.toml配置，继承官方全部检测规则 3. 配置pre-commit钩子，提交前自动扫描密钥硬编码风险。 ## Does this PR introduce any user-facing change? 无. ## How was this patch tested? PR流水线pre-commit检测新增敏感信息检测. See merge request: Ascend/MindSpeed-LLM!4843 | 2 个月前 |
| fix(fsdp2): guard indexer loss scaler against zero num_items_in_batch Signed-off-by: Rostellaria <xiaoshaoan@huawei.com> | 2 天前 |
| [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Co-authored-by: wujinyuan1<wujinyuan1@huawei.com> # message auto-generated for no-merge-commit merge: !4448 merge master into master [feat]Modify and improve the configuration of the pre-commit open-source code detection tool Created-by: wujinyuan1 Commit-by: wujinyuan1 Merged-by: ascend-robot Description: ## What this PR does / why we need it? pre-commit 工具规则配置修改：试运行阶段发现python bandit工具检测过严，超出昇腾编程规范。 此次修改重点修改bandit规则。 ## Does this PR introduce any user-facing change? 门禁codecheck检测规则精简，提升代码合入效率 ## How was this patch tested? 代码扫描工具配置，不涉及代码仓功能 See merge request: Ascend/MindSpeed-LLM!4448 | 4 个月前 |
| docs(fsdp2): fix documentation format issues and improve readability Co-authored-by: wangjiangben<wangjiangben@huawei.com> # message auto-generated for no-merge-commit merge: !4476 merge docs/fix-fsdp2-docs-format into master docs(fsdp2): fix documentation format issues and improve readability Created-by: wangjiangben Commit-by: wangjiangben Merged-by: ascend-robot Description: ## Summary Fix format errors in FSDP2 documentation and optimize document structure to improve readability and compliance with Markdown standards. **Format Fixes:** - Fix table format issues in quantization.md (missing header separators, column alignment errors) - Fix HTML entity syntax error in arguments.md ("ulysses" missing semicolon) - Remove extra blank lines in code blocks - Standardize indentation in example scripts **Structure Optimization:** - Unify list markers to standard Markdown - syntax - Optimize DTensor section hierarchy for better structure clarity - Convert reference links to proper Markdown link format - Split long paragraphs in quantization descriptions for better readability - Fix MD032 lint error (add blank line before list) **Files Changed:** - arguments.md: Fix HTML entity syntax - fsdp2_basic_features.md: Optimize structure hierarchy and list format - quantization.md: Fix table format, optimize description text See merge request: Ascend/MindSpeed-LLM!4476 | 4 个月前 |
| docs: add git submodule management for 3rdparty dependencies Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !4875 merge master into master docs: add git submodule management for 3rdparty dependencies Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an optional Git Submodule-based installation method to the MindSpeed LLM install guide, enabling unified version management of third-party dependencies (MindSpeed, FSDPTurbo, Megatron-LM). Background: Previously, users had to manually git clone and git checkout each dependency repository (steps 5-7 in "方式二：源码安装"), which made version tracking difficult and error-prone. This PR introduces 方式三：Git Submodule统一管理（可选） as an alternative approach. Detailed changes: Add .gitmodules and register three submodules under 3rdparty/: 3rdparty/MindSpeed — tracks remote master branch (latest) 3rdparty/FSDPTurbo — tracks remote main branch (latest) 3rdparty/Megatron-LM — pinned to tag core_v0.12.1 (commit a845aa7e1) ## Does this PR introduce any user-facing change? Yes, this PR introduces a new optional installation method documented in docs/zh/pytorch/training/install_guide.md. Users can now choose between: 方式二：源码安装 (existing, unchanged) — manual clone + checkout 方式三：Git Submodule统一管理（可选） (new) — git submodule update --init --recursive for one-command dependency setup No changes to existing installation flows. The new method is fully backward-compatible. Users who adopt the submodule approach should be aware that: MindSpeed and FSDPTurbo track remote latest by default; use git submodule update --init --remote to pull the latest commits. Megatron-LM is pinned to core_v0.12.1 and will not auto-update. The symlink-based install requires a Unix-like environment (Linux/macOS). On Windows, use pip3 install -e . within each submodule directory instead. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!4875 | 1 个月前 |
| feat(pytroch): Merge upgrade 018 base code Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4999 merge prepare_018 into master feat(pytroch): Merge upgrade 018 base code Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR serves as the foundational update for 018 adaptation. It adds the mindspeed_llm directory under ests and removes DPOFeature, MambaModel, HighAvailabilityFeature, ai_framework_feature, and layerwise_disaggregated_training_feature to facilitate the subsequent integration of code related to the 018 upgrade. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? This PR only merges the basic code and does not go through testing; it will be verified during subsequent adaptation. See merge request: Ascend/MindSpeed-LLM!4999 | 25 天前 |
| docs: update README and CONTRIBUTING.md Co-authored-by: xuwenyue<xuwenyue1@huawei.com> # message auto-generated for no-merge-commit merge: !4440 merge master into master docs: update README and CONTRIBUTING.md Created-by: xuwenyue Commit-by: xuwenyue Merged-by: ascend-robot Description: ## What this PR does / why we need it? update README, CONTRIBUTING.md, FAQ.md, features/README.md, performance_tuning.md ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4440 | 4 个月前 |
| update LICENSE. add license Signed-off-by: fengliangjun <fengliangjun@huawei.com> | 2 年前 |
| feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4878 merge fsdp2-msprobe into master feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds optional precision-data collection support for both FSDP2 and Megatron training backends. ## Does this PR introduce any user-facing change? Yes. The feature is disabled by default and does not affect existing training jobs unless explicitly enabled. For FSDP2, users can configure the features in the training YAML: yaml training: msprobe: true msprobe_config_path: /absolute/path/to/msprobe_config.json model_io_trace: true model_io_trace_config_path: /absolute/path/to/model_io_trace_config.json For Megatron, the corresponding arguments are: bash --msprobe \ --msprobe-config-path /absolute/path/to/msprobe_config.json \ --model-io-trace \ --model-io-trace-config-path /absolute/path/to/model_io_trace_config.json Related documentation: - docs/zh/pytorch/tools/msprobe.md - docs/zh/pytorch/tools/model_io_trace.md - docs/zh/pytorch/features/fsdp2/arguments.md ## How was this patch tested? After enabling the function, can the relevant files be generated after the training script is completed See merge request: Ascend/MindSpeed-LLM!4878 | 26 天前 |
| fix: migrate llama2 convert scripts to v2 Co-authored-by: wanggangguo<wanggangguo@huawei.com> # message auto-generated for no-merge-commit merge: !4700 merge debug5 into master fix: migrate llama2 convert scripts to v2 Created-by: isfrapples Commit-by: wanggangguo Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. Migrate gemma2, llama2, plm, qwen3 lora checkpoint scripts to v2. 2. Add phi3.5-moe support in convert_ckpt_v2.py and model_cfg.json. 3. Add missing add_qkv_bias config for phi3.5-moe in model_cfg.json. ## Does this PR introduce any user-facing change? 1. No, only internal example scripts are updated to use the new converter. 2. All existing interfaces and behaviors remain unchanged. ## How was this patch tested? 1. All migrated scripts share the same CLI pattern as existing v2 tests. 2. phi3.5-moe changes are validated by existing CI configurations. fixes#1580 See merge request: Ascend/MindSpeed-LLM!4700 | 2 个月前 |
| docs: update resource links and beta terminology Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !5022 merge master_link_0907 into master docs: update resource links and beta terminology Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: Fixes #1831 Fixes #1839 ## What this PR does / why we need it? 1. Replaces the unavailable Berkeley MMLU data.tar references in the Chinese and English READMEs and model-evaluation guides with the CAIS-hosted Hugging Face download. 2. Replaces the unavailable Megatron-LM SC'21 paper references in the Chinese and English virtual-pipeline-parallel guides with arXiv 2104.04473. 3. Updates README.md feature maturity labels and their explanatory note from Prototype to beta, while preserving the surrounding Markdown structure. ## Does this PR introduce any user-facing change? Yes. Documentation readers now receive working MMLU dataset and Megatron-LM paper links, and the Chinese README displays the updated beta maturity terminology. No runtime APIs, commands, defaults, or model behavior change. ## How was this patch tested? 1. git diff --check fffc206f8351d8baa40a39134598dac94efe6767..HEAD passed. 2. Changed-file pre-commit passed trailing-whitespace, end-of-file, large-file, merge-conflict, private-key, codespell, and typos checks. Python/C++-specific hooks were not applicable. The local gitleaks-offline-scan hook could not run because the repository-required ./gitleaks executable is absent. 3. HTTP HEAD validation returned 200 for both updated resource destinations. 4. GitCode remote Git Hooks Checking passed during the force-with-lease push. See merge request: Ascend/MindSpeed-LLM!5022 | 15 天前 |
| docs: update resource links and beta terminology Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !5022 merge master_link_0907 into master docs: update resource links and beta terminology Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: Fixes #1831 Fixes #1839 ## What this PR does / why we need it? 1. Replaces the unavailable Berkeley MMLU data.tar references in the Chinese and English READMEs and model-evaluation guides with the CAIS-hosted Hugging Face download. 2. Replaces the unavailable Megatron-LM SC'21 paper references in the Chinese and English virtual-pipeline-parallel guides with arXiv 2104.04473. 3. Updates README.md feature maturity labels and their explanatory note from Prototype to beta, while preserving the surrounding Markdown structure. ## Does this PR introduce any user-facing change? Yes. Documentation readers now receive working MMLU dataset and Megatron-LM paper links, and the Chinese README displays the updated beta maturity terminology. No runtime APIs, commands, defaults, or model behavior change. ## How was this patch tested? 1. git diff --check fffc206f8351d8baa40a39134598dac94efe6767..HEAD passed. 2. Changed-file pre-commit passed trailing-whitespace, end-of-file, large-file, merge-conflict, private-key, codespell, and typos checks. Python/C++-specific hooks were not applicable. The local gitleaks-offline-scan hook could not run because the repository-required ./gitleaks executable is absent. 3. HTTP HEAD validation returned 200 for both updated resource destinations. 4. GitCode remote Git Hooks Checking passed during the force-with-lease push. See merge request: Ascend/MindSpeed-LLM!5022 | 15 天前 |
| !2500 update notice Merge pull request !2500 from sunjunjie/master | 1 年前 |
| Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 14 天前 |
| Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 14 天前 |
| Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 14 天前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 1 个月前 |
| feature(pytorch): FSDP2 support hardware-adaptive execution Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4343 merge fsdp2_gpu into master feature(pytorch): FSDP2 support hardware-adaptive execution Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? FSDP2 support hardware-adaptive execution ## Does this PR introduce any user-facing change? No ## How was this patch tested? pipeline test passed See merge request: Ascend/MindSpeed-LLM!4343 | 5 个月前 |
| Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 14 天前 |
| Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 14 天前 |
| [pytorch][feature]Add RL data processing Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3646 merge fenzhi into master [pytorch][feature]Add RL data processing Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: add preprocess prmompt 修改模板路径 See merge request: Ascend/MindSpeed-LLM!3646 | 10 个月前 |
| refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Co-authored-by: dingzicha1997<dingzilin@huawei.com> # message auto-generated for no-merge-commit merge: !4871 merge remove-triton-mhc into master refactor: remove deepseek4 redundant Triton paths and align CSA/MHC CLI Created-by: dingzicha1997 Commit-by: dingzicha1997 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR simplifies the DeepSeek-V4 CSA and MHC implementations after the Ascend fused paths became stable. Main changes: - Remove redundant Triton MHC, Sinkhorn, RMSNorm, and SFA implementations. - Retain the Triton pre_only kernel required by the MHC head stage. - Keep eager implementations as fallback paths. - Rename G2-related components to DeepSeek-V4 CSA/HCA terminology. - Relocate MHC operators into the common ops directories. - Consolidate CSA eager attention implementations. - Align DeepSeek-V4 CLI arguments with Megatron and MindSpeed conventions. - Remove incompatible legacy RoPE options from all DeepSeek-V4 scripts. ## Does this PR introduce any user-facing change? Yes. DeepSeek-V4 CLI configurations need to be updated as follows: | Type | Previous option | New option / behavior | |---|---|---| | Renamed | --use-ascend-mhc | --use-fused-mhc | | Renamed | --g2-window-size 128 | --sliding-window-size 128 | | Removed | --rope-head-dim 64 | --qk-pos-emb-head-dim 64 | | Removed | --norm-eps 1e-6 | --norm-epsilon 1e-6 | | Renamed | --position-embedding-type g2 | --position-embedding-type deepseek4 | | Replaced | --use-g2-indexer-loss | --indexer-loss-coeff 1.0; use 0 for inference | | Removed | --use-triton-sfa | --use-sparse-flash-attn | | Removed | --use-g2-attention | No replacement required | | Removed | --use-triton-sinkhorn | Use fused MHC or the eager fallback | | Removed | --use-triton-rmsnorm-without-weight | Use fused MHC or the eager fallback | | Removed | --mhc-recompute | No replacement required | | Removed from V4 scripts | --use-fused-rotary-pos-emb | Incompatible with the DeepSeek-V4 RoPE path | | Removed from V4 scripts | --use-rotary-position-embeddings | Overrides the DeepSeek-V4 position embedding type | | Retained | --use-triton-mhc | Only controls the MHC-head pre_only kernel | | Behavior changed | --use-fused-lightning-indexer and --use-fused-lightning-indexer-loss | Must be enabled or disabled together; the framework automatically enables both if only one is provided | Recommended DeepSeek-V4 position embedding configuration: shell --position-embedding-type deepseek4 --qk-pos-emb-head-dim 64 Do not combine it with: shell --use-fused-rotary-pos-emb --use-rotary-position-embeddings ## How was this patch tested? - Checked all DeepSeek-V4 training, fine-tuning, inference, POC, and ST scripts. - Verified that removed CLI options have no remaining DeepSeek-V4 references. - Ran bash -n against all affected DeepSeek-V4 shell scripts. - Ran Python compilation and applicable code-quality checks for the modified Python modules. - Ran git diff --check. - Applicable local pre-commit hooks passed. The local Gitleaks hook could not run because the ./gitleaks binary was unavailable. - GitCode remote Git Hooks passed successfully. - Verified that importing the repository without Triton remains protected by the existing try/except logic. See merge request: Ascend/MindSpeed-LLM!4871 | 1 个月前 |
| fix(pytorch): Smoke test fix Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5033 merge fix_0910 into master fix(pytorch): Smoke test fix Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR fixes the smoke test error caused by PR 5001. ## Does this PR introduce any user-facing change? Test cases executed normally. ## How was this patch tested? Smoke test cases have been verified successfully. See merge request: Ascend/MindSpeed-LLM!5033 | 13 天前 |
| refactor(megatron):update coverage script Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4295 merge br_master_coverage_fix_0313 into master refactor(megatron):update coverage script Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: # 覆盖率分析脚本更新 ---- ## What this PR does / why we need it? 更新仓库run_coverage.sh脚本，修复覆盖率分析扫描文件不全的问题 ## Does this PR introduce any user-facing change? 不影响仓库基本功能，旨在优化仓库覆盖率分析功能 ## How was this patch tested? 已在蓝区机器上自测通过 See merge request: Ascend/MindSpeed-LLM!4295 | 6 个月前 |
| feat: third-party dependency reduction Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4887 merge requirements into master feat: third-party dependency reduction Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Non-essential Dependency Cleanup. ## Does this PR introduce any user-facing change? Remove irrelevant dependencies to reduce the difficulty for customers to get started with installation. ## How was this patch tested? CI and Nightly-CI-Component-pipeline_MindSpeed-LLM_master. See merge request: Ascend/MindSpeed-LLM!4887 | 1 个月前 |
| add rl ci Co-authored-by: fh_188<fenghui32@huawei.com> # message auto-generated for no-merge-commit merge: !3951 merge master_rl_ci into master [pytorch][feature]Adapt RL for vllm0110 and add CI Created-by: weixin_44917616 Commit-by: fh_188 Merged-by: ascend-robot Description: [pytorch][feature]Adapt RL for vllm0110 and add CI See merge request: Ascend/MindSpeed-LLM!3951 | 9 个月前 |
| [pytorch][build]update setup.py version and python support range master Co-authored-by: tichang<tichang@huawei.com> # message auto-generated for no-merge-commit merge: !4076 merge chore/update_setup.py_version_and_pyversion_info into master [pytorch][build]update setup.py version and python support range master Created-by: tichang Commit-by: tichang Merged-by: ascend-robot Description: chore(setup): update setup.py version and python support range 在master分支修改了setup.py中的版本信息和python支持版本 See merge request: Ascend/MindSpeed-LLM!4076 | 7 个月前 |
| feat: add dsv4 fsdp sh && performance optimization Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4859 merge master into master feat: add dsv4 fsdp sh && performance optimization Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? ### Background As large language model parameters continue to grow, new model architectures like DeepSeek V4 impose higher demands on distributed training. MindSpeed-LLM, as a distributed training suite for LLMs based on the Ascend NPU ecosystem, needs to continuously adapt to new models and optimize training performance. Currently, the repository does not provide a complete training script with performance optimizations for DeepSeek V4 using the FSDP (Fully Sharded Data Parallel) backend, which limits users' ability to efficiently train DeepSeek V4 on Ascend NPUs. ### Detailed Changes This PR introduces the following changes: - **Add DeepSeek V4 FSDP training script** – provides a ready‑to‑use startup script (dsv4 fsdp sh) for FSDP‑based distributed training of DeepSeek V4, reducing the effort required to adapt the model to the FSDP backend. - **Performance optimization** – performs targeted tuning for the DeepSeek V4 model architecture on the FSDP backend, including but not limited to: 1. Integrating the MHC fused operator. 2. Integrating Torch's fused AdamW optimizer. 3. Removing redundant transpose operations. 4. Integrating the SwiGLU_Limit fused operator. - **Training workflow validation** – ensures that the new script covers the basic pre‑training path, confirming correctness and stability. ## Does this PR introduce any user-facing change? Yes. This PR provides a new entry‑point script for DeepSeek V4 FSDP training. Users can directly use this script to launch distributed pre‑training tasks for DeepSeek V4 without manually writing complex distributed configurations. ## How was this patch tested? ### Functional Correctness Verification - Run the newly added dsv4 fsdp sh script on an Ascend NPU cluster to verify that DeepSeek V4 can start FSDP distributed training successfully. - Monitor training loss to ensure it converges as expected, confirming the correctness of the training logic. ### Performance Verification - Compare training throughput (tokens/s) and memory usage before and after the optimization. - Validate that the performance improvements meet the expected targets. See merge request: Ascend/MindSpeed-LLM!4859 | 1 个月前 |
