| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: add dsv4 fsdp sh && performance optimization Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4859 merge master into master feat: add dsv4 fsdp sh && performance optimization Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? ### Background As large language model parameters continue to grow, new model architectures like DeepSeek V4 impose higher demands on distributed training. MindSpeed-LLM, as a distributed training suite for LLMs based on the Ascend NPU ecosystem, needs to continuously adapt to new models and optimize training performance. Currently, the repository does not provide a complete training script with performance optimizations for DeepSeek V4 using the FSDP (Fully Sharded Data Parallel) backend, which limits users' ability to efficiently train DeepSeek V4 on Ascend NPUs. ### Detailed Changes This PR introduces the following changes: - **Add DeepSeek V4 FSDP training script** – provides a ready‑to‑use startup script ( dsv4 fsdp sh) for FSDP‑based distributed training of DeepSeek V4, reducing the effort required to adapt the model to the FSDP backend. - **Performance optimization** – performs targeted tuning for the DeepSeek V4 model architecture on the FSDP backend, including but not limited to: 1. Integrating the MHC fused operator. 2. Integrating Torch's fused AdamW optimizer. 3. Removing redundant transpose operations. 4. Integrating the SwiGLU_Limit fused operator. - **Training workflow validation** – ensures that the new script covers the basic pre‑training path, confirming correctness and stability. ## Does this PR introduce any user-facing change? Yes. This PR provides a new entry‑point script for DeepSeek V4 FSDP training. Users can directly use this script to launch distributed pre‑training tasks for DeepSeek V4 without manually writing complex distributed configurations. ## How was this patch tested? ### Functional Correctness Verification - Run the newly added dsv4 fsdp sh script on an Ascend NPU cluster to verify that DeepSeek V4 can start FSDP distributed training successfully. - Monitor training loss to ensure it converges as expected, confirming the correctness of the training logic. ### Performance Verification - Compare training throughput (tokens/s) and memory usage before and after the optimization. - Validate that the performance improvements meet the expected targets. See merge request: Ascend/MindSpeed-LLM!4859 | 9 小时前 | |
feat: FSDP2 Parameter Extraction from YAML to Shell Scripts Co-authored-by: HANHU1CHEN<chenhanhui1@h-partners.com> # message auto-generated for no-merge-commit merge: !4722 merge master into master feat: FSDP2 Parameter Extraction from YAML to Shell Scripts Created-by: Rostellaria Commit-by: HANHU1CHEN Merged-by: ascend-robot Description: [#1586](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1586) ## What this PR does / why we need it? Refactors FSDP2 training/inference scripts: extracts parallel , training , optimization , and quantization parameters from YAML configs into shell scripts as inline torchrun CLI arguments, enabling easier per-run overrides without editing YAML files. Keeps data.dataset.file_name in YAML due to argparse Dict-type field constraints. ## Does this PR introduce any user-facing change? Yes. Users now configure parallelism, batch size, output dir, and optimization params directly in .sh scripts instead of .yaml files. YAML files only retain model, data, and recompute configs. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Ran pretrain_qwen3_8b_4k_fsdp2.sh in docker with conda env, completed 15 training steps successfully (exit code 0). See merge request: Ascend/MindSpeed-LLM!4722 | 26 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feat: FSDP2 Longcat-Flash-Lite adaptation for the GEMM function Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4753 merge fsdp2-gemm into master feat: FSDP2 Longcat-Flash-Lite adaptation for the GEMM function Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? LongCat-Flash-Lite supports online weight conversion, Grouped GEMM (GMM) expert computation, and N-gram embedding memory optimization in the FSDP2 scenario. ## Does this PR introduce any user-facing change? The model can directly use the original HuggingFace weights to start FSDP2 training/inference, and reduce the risk of out-of-memory (OOM) caused by dense gradient accumulation in the backward phase of multiple N-gram embedding tables. ## How was this patch tested? Single-machine layer reduction precision alignment, with an absolute error of 0.004. See merge request: Ascend/MindSpeed-LLM!4753 | 9 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feat(fsdp2): add Qwen3-1.7B pretraining and fine-tuning recipes for Atlas A2 Co-authored-by: lishuai26<lishuai_math@163.com> # message auto-generated for no-merge-commit merge: !4754 merge qwen3-1.7b-fsdp2 into master feat(fsdp2): add Qwen3-1.7B pretraining and fine-tuning recipes for Atlas A2 Created-by: lishuai26 Commit-by: lishuai26 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Related to #1352. This PR adds FSDP2 pretraining and full-parameter fine-tuning recipes for Qwen3-1.7B on a single node with 4 Atlas 800T A2 NPUs. The existing Qwen3 FSDP2 examples mainly target the Qwen3-8B model on 8 NPUs. This patch provides a Qwen3-1.7B recipe adapted for a 4-NPU environment and a sequence length of 4096. The following files are added: - examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.sh - examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.yaml - examples/fsdp2/qwen3/tune_qwen3_1p7b_4k_fsdp2_A2.sh - examples/fsdp2/qwen3/tune_qwen3_1p7b_4k_fsdp2_A2.yaml The main changes include: 1. Add a single-node, 4-NPU FSDP2 launch configuration for Qwen3-1.7B. 2. Add a 4K sequence-length pretraining configuration using the Megatron-style data manager. 3. Add a full-parameter SFT configuration using the Alpaca data format and the default LF data manager. 4. Enable activation recomputation for model.layers.{*} to reduce activation memory usage. 5. Enable throughput and MFU logging for performance evaluation. 6. Pass the fused RMSNorm, fused RoPE, and FlashAttention optimization switches through the launch scripts. 7. Allow MODEL_PATH, OUTPUT_DIR, and additional CLI arguments to override the default configuration without editing the scripts. 8. Save timestamped training logs for reproducibility and troubleshooting. ## Does this PR introduce any user-facing change? Yes. This PR adds new runnable examples for users who want to pretrain or fine-tune Qwen3-1.7B with the MindSpeed-LLM FSDP2 backend on 4 Atlas 800T A2 NPUs. Example usage: bash # pretraining MODEL_PATH=/path/to/Qwen3-1.7B \ OUTPUT_DIR=./output/qwen3_1p7b_pretrain \ bash examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.sh # fine-tuning MODEL_PATH=/path/to/Qwen3-1.7B \ OUTPUT_DIR=./output/qwen3_1p7b_sft \ bash examples/fsdp2/qwen3/tune_qwen3_1p7b_4k_fsdp2_A2.sh Users must replace the model and dataset paths with paths available in their own environment. This patch does not change existing APIs, the FSDP2 training core, or existing model recipes. The related example paths are: - examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.sh - examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.yaml - examples/fsdp2/qwen3/tune_qwen3_1p7b_4k_fsdp2_A2.sh - examples/fsdp2/qwen3/tune_qwen3_1p7b_4k_fsdp2_A2.yaml ## How was this patch tested? The pretraining recipe was smoke-tested on a single node with 4 Atlas 800T A2 NPUs. Test configuration: - Model: Qwen3-1.7B - Backend: FSDP2 - Sequence length: 4096 - Per-device batch size: 4 - Global batch size: 16 - Activation recomputation: enabled - Gradient reduction dtype: FP32 - Test steps: 100 Command: bash bash examples/fsdp2/qwen3/pretrain_qwen3_1p7b_4k_fsdp2_A2.sh \ --training.max_steps 100 \ --training.output_dir ./output/qwen3_1p7b_smoke_test See merge request: Ascend/MindSpeed-LLM!4754 | 17 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 19 天前 | |
feature(pytorch): FSDP2 support hardware-adaptive execution Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4343 merge fsdp2_gpu into master feature(pytorch): FSDP2 support hardware-adaptive execution Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? FSDP2 support hardware-adaptive execution ## Does this PR introduce any user-facing change? No ## How was this patch tested? pipeline test passed See merge request: Ascend/MindSpeed-LLM!4343 | 4 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 9 小时前 | ||
| 26 天前 | ||
| 19 天前 | ||
| 9 天前 | ||
| 19 天前 | ||
| 19 天前 | ||
| 17 天前 | ||
| 19 天前 | ||
| 19 天前 | ||
| 19 天前 | ||
| 4 个月前 |