| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: update ST baselines Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4861 merge master into master feat: update ST baselines Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1696](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1696) ## 1. What this PR does / why we need it? This commit refreshes the performance baselines for 6 ST (system-test) cases, fixes a Docker issue where CANN's built-in Python lacked Python.h / python3-config to compile C++ extensions, and removes the deprecated --moe-permute-fusion flag from deepseek4_flash_mcore_tp1_pp1_ep8.sh . It is needed because recent code changes (e.g. MoE logic) shifted per-iteration timing, making the old baselines unsuitable for accurate regression comparison. ## 2. Does this PR introduce any user-facing change? No end-user-facing changes; it only affects CI/ST test baselines and the comparison logic in tests/test_tools/test_ci_st.py , where _compare_time was simplified by removing the DROP_MAX_TIME_CASES special handling that dropped the maximum time for a single case. No user documentation needs updating. ## 3. How was this patch tested? Verified by re-running the affected ST cases (e.g. qwen3_30b_tp4_cp2_ep2 , pretrain_qwen3_8b_4k_hybridcp_gqa ) and comparing actual timings against the new baselines using the updated test_ci_st.py (averaging iterations after a 5-step warmup). Limitation: the timing baselines are hardware/image-specific and must be regenerated when the environment changes. See merge request: Ascend/MindSpeed-LLM!4861 | 1 个月前 | |
test(python): update dpo st Co-authored-by: yanzhixiao<yanzhixiao@h-partners.com> # message auto-generated for no-merge-commit merge: !4563 merge update-dpo-st into master test(python): update dpo st Created-by: yanzhixiao23 Commit-by: yanzhixiao Merged-by: ascend-robot Description: ## What this PR does / why we need it? update dpo st using qwen3-30B ## Does this PR introduce any user-facing change? Na ## How was this patch tested? Na See merge request: Ascend/MindSpeed-LLM!4563 | 2 个月前 | |
test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4963 merge master into master test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1784](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1784) ## What this PR does / why we need it? This PR sinks two redundant or long-running gate cases (pretrain_deepseek4_flash_1k_fsdp2, deepseek_v3_mcore_tp1_pp2_ep4) into the nightly pipeline to shorten gate CI time, as their guarded features are fully covered by remaining gate cases. It also fixes the mamba3 pipeline script, whose tee into a non-existent logs/ directory falsely failed the case with "Script has failed" in clean CI checkouts. ## Does this PR introduce any user-facing change? No. This PR only reassigns test cases between the gate and pipeline suites and updates the registrations in tests/README.md and tests/resource_record.md; no framework code or user-facing usage changes are involved. ## How was this patch tested? All changed scripts passed bash -n syntax checks, the gate table now matches the 11 remaining .sh files one-to-one, and mamba3 was re-run 3 times on 8 NPUs with loss/grad-norm matching baseline values. The moved cases are still auto-discovered and executed nightly by the pipeline runner. See merge request: Ascend/MindSpeed-LLM!4963 | 19 天前 | |
feat(torch): deepseekv4 cp feat Co-authored-by: mhh111<mahonghao1@huawei.com> # message auto-generated for no-merge-commit merge: !4855 merge deepseek_v4_cp_sync into master feat(torch): deepseekv4 cp feat Created-by: mhh111 Commit-by: mhh111 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Background: DeepSeek-V4 on FSDP2 uses sparse attention (CSA/HCA compressor + Lightning Indexer + the fused npu_sparse_flash_mla op). The existing CP schemes (ulysses/ring) do not fit directly: the fused sparse-flash op applies its causal+sliding mask relative to the query index with no query-position offset, and derives cmp_residual_kv from the query length. Feeding a local query shard would break causality on every rank except rank 0 and compute the wrong cmp_residual_kv. Approach: Add cp_type="kvallgather" for DeepSeek-V4. Each CP rank holds a local sequence shard [local_start:local_end]; q/kv and compressor inputs are all-gathered (via a differentiable all-gather) into the full sequence, every rank runs the same attention dispatch as non-CP on the full sequence, then slices out its local rows. KV-allgather keeps the op semantics bit-for-bit aligned with non-CP. Detailed changes (base: master 4332172d; 1 new + 8 modified): - deepseek_v4_cp_attention.py (new): CP attention forward. Differentiable all-gather of q/kv/compressor inputs; runs the identical dispatch as master (fuse_indexer_loss / use_sparse_flash_attn / dense) on the full sequence and slices the local rows; the indexer KL loss reuses master's DeepseekV4Attention._compute_indexer_kl_loss + IndexerLossAutoScaler (q/kv detached inside), so the loss value and gradient routing match CP-off. - context_parallel_mappings.py: register the deepseek_v4_kvallgather mapping (DeepseekV4Attention.forward → deepseek_v4_cp_attention_forward). - context_parallel_manager.py / mindspeed_parallel_engine.py / parallel_engine_config.py / arguments.py: accept cp_type="kvallgather". - trainer.py: backward loss = loss * cp_size (loss_for_log keeps the pre-scale value so CP-on and CP-off report the same loss magnitude); kvallgather reuses the ulysses batch-sampling path; dataset_id is skipped when CP-sharding. - clip_grad_norm.py: drop total_norm *= cp_size (CP scaling is now handled uniformly by loss*cp_size in the trainer; the EP-aware ep_fsdp2_clip_grad_norm path is untouched). - mtp.py: roll_tensor gains a cp_group arg; CP boundaries exchange the boundary token with the adjacent rank via P2P so the MTP next-token target is correct across CP boundaries. CP consistency: CP-on computes attention on the full sequence and slices local rows, so each local output equals the corresponding row of the non-CP output; the indexer loss is computed on the full (all-gatheredce, identical to CP-off; the reported loss uses loss_for_log(pre-scale). ## Does this PR introduce any user-facing change? Yes. Adds cp_type="kvallgather" for DeepSeek-V4 FSDP2 training. - Usage: set cp_type: kvallgather and cp_size: N (N>1) in the parallel arguments. This PR does not bundle example configs; users set these in their DeepSeek-V4 training config. - Behavioral change (affects all cp types): CP gradient scaling is unified from "total_norm *= cp_size in clip_grad_norm" to "loss = loss * cp_size in the trainer". ulysses/ring previously used grad-norm scaling and nkvallgather); this moves the scaling point from the clipthreshold to the loss and is numerically equivalent for the CP gradient. - Constraints: cp_size must divide the sequence length; the KVl KV per rank and adds q/kv/hidden all-gather communication perlayer, so it suits moderate cp_size. - No doc changes (an fsdp2 CP usage doc can be added in a follow-up if needed). ## How was this patch tested? - Static checks: all 9 touched Python files pass ast syntax parsing; no conflict markers; master files (modeling/op/indexer_loss) are left unchanged (verified via git diff). - Logic review: the CP forward's dispatch branches align item-ion.forward (same needs_indexer_loss/fuse_indexer_loss checks,same _sparse_flash_attn[_with_indexer_loss], same _compute_indexer_kl_loss), so each token's output equals the corresponding row of the non-CP output. - Recommended NPU verification (to be run on Ascend): with a DeepSeek-V4 pretraining config, compare cp_size=1 (CP-off) vs cp_size=2 (CP-on, cp_type=kvallgather) loss and a few steps of gradient norm — tlerance. - Limitations & caveats: - fused+CP (use_fused_lightning_indexer_loss=True): supportebranch); each rank redundantly computes the full-sequenceindexer loss (same per-rank behavior as the non-fused CP path). - dense (eager) fallback (use_sparse_flash_attn=False): CP bmask; verify separately if this path is used (default usessparse_flash). - MTP loss: the trainer-side roll_tensor(labels) recomputation of MTP loss does not pass cp_group (pre-existing master behavior, same for ulysses/ring); the CP-boundary token's MTP label is set to -100 (1 token per CP boundary, <0.05% effect on MTP loss, negligible). - This branch is based on master 4332172d and has not been rebased onto the latest a2b5ba0c; a rebase can be done as a follow-up if needed. https://gitcode.com/Ascend/MindSpeed-LLM/issues/1670 See merge request: Ascend/MindSpeed-LLM!4855 | 1 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 1 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 1 个月前 | |
test(megatron):replace chatglm3_gqa_cp4 testcase with qwen3-8b Co-authored-by: guozhihua2<guozhihua2@huawei.com> # message auto-generated for no-merge-commit merge: !4511 merge remove_chatglm3_master into master test(megatron):replace chatglm3_gqa_cp4 testcase with qwen3-8b Created-by: guozhihua2 Commit-by: guozhihua2 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Replace chatglm3_gqa_cp4 testcase with qwen3-8b in st and move old case to pipeline. ## Does this PR introduce any user-facing change? Only test cases are replaced, and service code is not modified. ## How was this patch tested? test by CI. See merge request: Ascend/MindSpeed-LLM!4511 | 2 个月前 | |
feat: change ST Co-authored-by: zhyebin01<zhangyebin@h-partners.com> # message auto-generated for no-merge-commit merge: !4567 merge st into master feat: change ST Created-by: zhyebin01 Commit-by: zhyebin01 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 替换原有的mixtral st用例，看护相同功能项 ## Does this PR introduce any user-facing change? NA. ## How was this patch tested? 当前PR即为ST用例 See merge request: Ascend/MindSpeed-LLM!4567 | 3 个月前 | |
feat(torch):add st Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4492 merge master into master feat(torch):add st Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-LLM!4492 | 3 个月前 | |
feat: change ST's llama2_tp4pp2vpp2_tp2d_tpx2tpy2.sh to qwen3_8b_tp2_pp4_vpp2.sh Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !4562 merge master_st into master feat: change ST's llama2_tp4pp2vpp2_tp2d_tpx2tpy2.sh to qwen3_8b_tp2_pp4_vpp2.sh Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: ## What this PR does / why we need it? 更换原有的llama2对应的ST用例，看护相同功能项. ## Does this PR introduce any user-facing change? NA. ## How was this patch tested? 当前PR即为ST用例. See merge request: Ascend/MindSpeed-LLM!4562 | 2 个月前 | |
test(megatron): replace llama2_tp2_pp4_vpp2_swap testcase with qwen3-8b Co-authored-by: tichang<tichang@huawei.com> # message auto-generated for no-merge-commit merge: !4508 merge st/replace_swap_st into master test(megatron): replace llama2_tp2_pp4_vpp2_swap testcase with qwen3-8b Created-by: tichang Commit-by: tichang Merged-by: ascend-robot Description: ## What this PR does / why we need it? Please describe the background and detailed changes of the PR. If it is a bugfix, please attach the related issue. ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-LLM!4508 | 2 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 1 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 1 个月前 | |
test(megatron): replace_llama3_lora_with_qwen3_8b Co-authored-by: JZY_SC30<jiangzhiyou@huawei.com> # message auto-generated for no-merge-commit merge: !4454 merge ci_update into master test(megatron): replace_llama3_lora_with_qwen3_8b Created-by: JZY_SC30 Commit-by: JZY_SC30 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Uninstall the old test case and replace it with the new one. ## Does this PR introduce any user-facing change? Only test cases are replaced, and service code is not modified. ## How was this patch tested? test by CI See merge request: Ascend/MindSpeed-LLM!4454 | 3 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 2 个月前 | ||
| 19 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 3 个月前 |