| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4878 merge fsdp2-msprobe into master feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds optional precision-data collection support for both FSDP2 and Megatron training backends. ## Does this PR introduce any user-facing change? Yes. The feature is disabled by default and does not affect existing training jobs unless explicitly enabled. For FSDP2, users can configure the features in the training YAML: yaml training: msprobe: true msprobe_config_path: /absolute/path/to/msprobe_config.json model_io_trace: true model_io_trace_config_path: /absolute/path/to/model_io_trace_config.json For Megatron, the corresponding arguments are: bash --msprobe \ --msprobe-config-path /absolute/path/to/msprobe_config.json \ --model-io-trace \ --model-io-trace-config-path /absolute/path/to/model_io_trace_config.json Related documentation: - docs/zh/pytorch/tools/msprobe.md - docs/zh/pytorch/tools/model_io_trace.md - docs/zh/pytorch/features/fsdp2/arguments.md ## How was this patch tested? After enabling the function, can the relevant files be generated after the training script is completed See merge request: Ascend/MindSpeed-LLM!4878 | 1 个月前 | |
test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4963 merge master into master test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1784](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1784) ## What this PR does / why we need it? This PR sinks two redundant or long-running gate cases (pretrain_deepseek4_flash_1k_fsdp2, deepseek_v3_mcore_tp1_pp2_ep4) into the nightly pipeline to shorten gate CI time, as their guarded features are fully covered by remaining gate cases. It also fixes the mamba3 pipeline script, whose tee into a non-existent logs/ directory falsely failed the case with "Script has failed" in clean CI checkouts. ## Does this PR introduce any user-facing change? No. This PR only reassigns test cases between the gate and pipeline suites and updates the registrations in tests/README.md and tests/resource_record.md; no framework code or user-facing usage changes are involved. ## How was this patch tested? All changed scripts passed bash -n syntax checks, the gate table now matches the 11 remaining .sh files one-to-one, and mamba3 was re-run 3 times on 8 NPUs with loss/grad-norm matching baseline values. The moved cases are still auto-discovered and executed nightly by the pipeline runner. See merge request: Ascend/MindSpeed-LLM!4963 | 2 个月前 | |
test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4963 merge master into master test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1784](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1784) ## What this PR does / why we need it? This PR sinks two redundant or long-running gate cases (pretrain_deepseek4_flash_1k_fsdp2, deepseek_v3_mcore_tp1_pp2_ep4) into the nightly pipeline to shorten gate CI time, as their guarded features are fully covered by remaining gate cases. It also fixes the mamba3 pipeline script, whose tee into a non-existent logs/ directory falsely failed the case with "Script has failed" in clean CI checkouts. ## Does this PR introduce any user-facing change? No. This PR only reassigns test cases between the gate and pipeline suites and updates the registrations in tests/README.md and tests/resource_record.md; no framework code or user-facing usage changes are involved. ## How was this patch tested? All changed scripts passed bash -n syntax checks, the gate table now matches the 11 remaining .sh files one-to-one, and mamba3 was re-run 3 times on 8 NPUs with loss/grad-norm matching baseline values. The moved cases are still auto-discovered and executed nightly by the pipeline runner. See merge request: Ascend/MindSpeed-LLM!4963 | 2 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 3 个月前 | |
fix(pytorch):add ckpt-format argument to scripts Co-authored-by: z__y<z4t155664@163.com> # message auto-generated for no-merge-commit merge: !4371 merge add_ckpt_torch_dist_argument_for_shells into master fix(pytorch):add ckpt-format argument to scripts Created-by: z__y Commit-by: z__y Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR explicitly adds ckpt-format torch to all repository scripts to support the asynchronous checkpoint saving feature. ## Does this PR introduce any user-facing change? No. This change only adjusts internal script parameters to maintain existing behavior. There are no user-facing API or usage changes. ## How was this patch tested? Tests confirm that asynchronous checkpoint saving works correctly and that the original torch format checkpoint behavior is preserved. See merge request: Ascend/MindSpeed-LLM!4371 | 6 个月前 | |
feat(pytorch): add hy3 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4781 merge master into master feat(pytorch): add hy3 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? - Add Hunyuan3 model adaptation. - Add Hunyuan3 model configuration. - Add Hunyuan3 weight conversion support. - Integrate Hunyuan3 into the existing model loading and training workflow. ## Does this PR introduce any user-facing change? Yes. This PR introduces support for the Hunyuan3 model. Users can now use MindSpeed-LLM to perform weight conversion and training for Hunyuan3 following the existing workflow. No user-facing changes are introduced for existing supported models. ## How was this patch tested? The following tests have been completed: - Pipeline ST passed. - Verified that Hunyuan3 weight conversion completes successfully. - Verified that the converted checkpoint can be loaded successfully. - Verified that the training workflow starts successfully after conversion. See merge request: Ascend/MindSpeed-LLM!4781 | 2 个月前 | |
fix(pytorch):add ckpt-format argument to scripts Co-authored-by: z__y<z4t155664@163.com> # message auto-generated for no-merge-commit merge: !4371 merge add_ckpt_torch_dist_argument_for_shells into master fix(pytorch):add ckpt-format argument to scripts Created-by: z__y Commit-by: z__y Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR explicitly adds ckpt-format torch to all repository scripts to support the asynchronous checkpoint saving feature. ## Does this PR introduce any user-facing change? No. This change only adjusts internal script parameters to maintain existing behavior. There are no user-facing API or usage changes. ## How was this patch tested? Tests confirm that asynchronous checkpoint saving works correctly and that the original torch format checkpoint behavior is preserved. See merge request: Ascend/MindSpeed-LLM!4371 | 6 个月前 | |
fix: fix mg2hf bug, support qkv_split per_group layout for seed-oss mcore2hf conversion Co-authored-by: wanggangguo<wanggangguo@huawei.com> # message auto-generated for no-merge-commit merge: !4656 merge upgrade into master fix: fix mg2hf bug, support qkv_split per_group layout for seed-oss mcore2hf conversion Created-by: isfrapples Commit-by: wanggangguo Merged-by: ascend-robot Description: ## What this PR does / why we need it? 1. Adds seed-oss and qwen25 to the v2 converter's supported model list with proper model_cfg.json configuration. 2. Fixes fused QKV weight splitting by introducing a qkv_split attribute so the converter correctly handles both per-group-interleaved (seed-oss) and contiguous (all other models) memory layouts. 3. Fixes qwen25 conversion producing checkpoints without linear_qkv.bias by registering it as a standalone model type instead of falling back to llama2. ## Does this PR introduce any user-facing change? 1. **seed-oss**: mcore2hf conversion now produces correct Q/K/V weights. 2. **qwen25**: existing checkpoints converted with --model-type-hf llama2 must be re-converted with --model-type-hf qwen25 to include linear_qkv.bias. 3. **All other models**: no behavioral change. ## How was this patch tested? 1. All 5 mcore2hf cases covering qkv_type='unpack' (seed-oss MHA TP2, qwen25 MHA TP4, qwen2-moe GQA, llama2 MQA TP2, llama2 MHA TP1) passed. 2. HF→MG round-trip cases for qwen25 and qwen2-moe passed. 3. Four qwen25 layerwise disaggregated training CI scripts were updated to --model-type-hf qwen25, fixing the previously reported Missing linear_qkv.bias errors. fixes#1551 See merge request: Ascend/MindSpeed-LLM!4656 | 3 个月前 | |
feat: FSDP2 Longcat-Flash-Lite adaptation for the GEMM function Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4753 merge fsdp2-gemm into master feat: FSDP2 Longcat-Flash-Lite adaptation for the GEMM function Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? LongCat-Flash-Lite supports online weight conversion, Grouped GEMM (GMM) expert computation, and N-gram embedding memory optimization in the FSDP2 scenario. ## Does this PR introduce any user-facing change? The model can directly use the original HuggingFace weights to start FSDP2 training/inference, and reduce the risk of out-of-memory (OOM) caused by dense gradient accumulation in the backward phase of multiple N-gram embedding tables. ## How was this patch tested? Single-machine layer reduction precision alignment, with an absolute error of 0.004. See merge request: Ascend/MindSpeed-LLM!4753 | 2 个月前 | |
fix(pytorch):add ckpt-format argument to scripts Co-authored-by: z__y<z4t155664@163.com> # message auto-generated for no-merge-commit merge: !4371 merge add_ckpt_torch_dist_argument_for_shells into master fix(pytorch):add ckpt-format argument to scripts Created-by: z__y Commit-by: z__y Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR explicitly adds ckpt-format torch to all repository scripts to support the asynchronous checkpoint saving feature. ## Does this PR introduce any user-facing change? No. This change only adjusts internal script parameters to maintain existing behavior. There are no user-facing API or usage changes. ## How was this patch tested? Tests confirm that asynchronous checkpoint saving works correctly and that the original torch format checkpoint behavior is preserved. See merge request: Ascend/MindSpeed-LLM!4371 | 6 个月前 | |
test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4963 merge master into master test: sink deepseek4-flash fsdp2 and deepseek-v3 gate cases to pipeline to shorten CI time, fix mamba3 tee misreport Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1784](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1784) ## What this PR does / why we need it? This PR sinks two redundant or long-running gate cases (pretrain_deepseek4_flash_1k_fsdp2, deepseek_v3_mcore_tp1_pp2_ep4) into the nightly pipeline to shorten gate CI time, as their guarded features are fully covered by remaining gate cases. It also fixes the mamba3 pipeline script, whose tee into a non-existent logs/ directory falsely failed the case with "Script has failed" in clean CI checkouts. ## Does this PR introduce any user-facing change? No. This PR only reassigns test cases between the gate and pipeline suites and updates the registrations in tests/README.md and tests/resource_record.md; no framework code or user-facing usage changes are involved. ## How was this patch tested? All changed scripts passed bash -n syntax checks, the gate table now matches the 11 remaining .sh files one-to-one, and mamba3 was re-run 3 times on 8 NPUs with loss/grad-norm matching baseline values. The moved cases are still auto-discovered and executed nightly by the pipeline runner. See merge request: Ascend/MindSpeed-LLM!4963 | 2 个月前 | |
fix(pytorch): minimax-m27 pipeline fix Co-authored-by: HanhuiChen<chenhanhui1@h-partners.com> # message auto-generated for no-merge-commit merge: !5073 merge bugfix into master fix(pytorch): minimax-m27 pipeline fix Created-by: HANHU1CHEN Commit-by: HanhuiChen Merged-by: ascend-robot Description: ## What this PR does / why we need it? Fix tokenizer loading in the data preprocessing pipeline for MiniMax-M2 and Step-3.5 models. For these two models, loading the tokenizer via AutoTokenizer.from_pretrained does not work correctly. This PR reads the model config with AutoConfig in _AutoTokenizer. When model_type is minimax_m2 or step3p5, the tokenizer is loaded explicitly with GPT2Tokenizer. All other models keep using AutoTokenizer as before. ## Does this PR introduce any user-facing change? No. Usage and arguments are unchanged; the tokenizer class is selected automatically based on model_type. ## How was this patch tested? - Ran data preprocessing with MiniMax-M2 and Step-3.5 weights; the tokenizer loads successfully and the encode/decode results match the original HF tokenizer. - Ran data preprocessing with an existing model (e.g. Qwen / DeepSeek) to confirm the AutoTokenizer path is unaffected. Limitation: tokenizer_name_or_path must contain a valid config.json, since AutoConfig is loaded with local_files_only=True. See merge request: Ascend/MindSpeed-LLM!5073 | 25 天前 | |
fix(pytorch):add ckpt-format argument to scripts Co-authored-by: z__y<z4t155664@163.com> # message auto-generated for no-merge-commit merge: !4371 merge add_ckpt_torch_dist_argument_for_shells into master fix(pytorch):add ckpt-format argument to scripts Created-by: z__y Commit-by: z__y Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR explicitly adds ckpt-format torch to all repository scripts to support the asynchronous checkpoint saving feature. ## Does this PR introduce any user-facing change? No. This change only adjusts internal script parameters to maintain existing behavior. There are no user-facing API or usage changes. ## How was this patch tested? Tests confirm that asynchronous checkpoint saving works correctly and that the original torch format checkpoint behavior is preserved. See merge request: Ascend/MindSpeed-LLM!4371 | 6 个月前 | |
feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4878 merge fsdp2-msprobe into master feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds optional precision-data collection support for both FSDP2 and Megatron training backends. ## Does this PR introduce any user-facing change? Yes. The feature is disabled by default and does not affect existing training jobs unless explicitly enabled. For FSDP2, users can configure the features in the training YAML: yaml training: msprobe: true msprobe_config_path: /absolute/path/to/msprobe_config.json model_io_trace: true model_io_trace_config_path: /absolute/path/to/model_io_trace_config.json For Megatron, the corresponding arguments are: bash --msprobe \ --msprobe-config-path /absolute/path/to/msprobe_config.json \ --model-io-trace \ --model-io-trace-config-path /absolute/path/to/model_io_trace_config.json Related documentation: - docs/zh/pytorch/tools/msprobe.md - docs/zh/pytorch/tools/model_io_trace.md - docs/zh/pytorch/features/fsdp2/arguments.md ## How was this patch tested? After enabling the function, can the relevant files be generated after the training script is completed See merge request: Ascend/MindSpeed-LLM!4878 | 1 个月前 | |
test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4909 merge master into master test: move qwen3_8b cp_ring test to pipeline and update deepseek4 flash ckpt path Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1704](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1704) ## What this PR does / why we need it? This PR moves the pretrain_qwen3_8b_general_cp_ring test script and its baseline JSON from the st directory to the pipeline directory to align with the pipeline test conventions. It also updates the deepseek4-flash checkpoint path and adds timestamped log output. ## Does this PR introduce any user-facing change? No user-facing change. The README test index table is updated to reflect the new path, but there is no impact on end-user functionality. ## How was this patch tested? The qwen3-8b cp_ring and deepseek4-flash scripts were executed successfully on an 8-NPU environment, and the results were verified against baselines. See merge request: Ascend/MindSpeed-LLM!4909 | 2 个月前 | |
fix(pytorch): fix bug of torch.mm Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !5011 merge master into master fix(pytorch): fix bug of torch.mm Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR fixes the logits computation by transposing head_weight before matrix multiplication. python logits = torch.mm(hidden_states, head_weight.T).float() Without the transpose, the matrix dimensions are incompatible with the layout of the LM head weight. ## Does this PR introduce any user-facing change? No. This is an internal bug fix and does not change any user-facing APIs or configurations. ## How was this patch tested? Verified that logits can be computed successfully and that the output shape is correct. See merge request: Ascend/MindSpeed-LLM!5011 | 1 个月前 | |
fix(pytorch):add ckpt-format argument to scripts Co-authored-by: z__y<z4t155664@163.com> # message auto-generated for no-merge-commit merge: !4371 merge add_ckpt_torch_dist_argument_for_shells into master fix(pytorch):add ckpt-format argument to scripts Created-by: z__y Commit-by: z__y Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR explicitly adds ckpt-format torch to all repository scripts to support the asynchronous checkpoint saving feature. ## Does this PR introduce any user-facing change? No. This change only adjusts internal script parameters to maintain existing behavior. There are no user-facing API or usage changes. ## How was this patch tested? Tests confirm that asynchronous checkpoint saving works correctly and that the original torch format checkpoint behavior is preserved. See merge request: Ascend/MindSpeed-LLM!4371 | 6 个月前 | |
feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742 | 3 个月前 | |
test(megatron): CI run added pipeline testcase Co-authored-by: guihaowen666<guihaowen@huawei.com> # message auto-generated for no-merge-commit merge: !4670 merge feature/pipeline-case-gate into master test(megatron): CI run added pipeline testcase Created-by: guihaowen666 Commit-by: guihaowen666 Merged-by: ascend-robot Description: ## What this PR does / why we need it? When adding or modifying pipeline test cases, not only are UT and ST test cases executed, but the newly added or modified pipeline test cases are also tested. This prevents pipeline test cases that would cause merge failures from being merged. ## Does this PR introduce any user-facing change? The access control architecture is modified, but the service code is not modified. Therefore, the user plane is not affected. ## How was this patch tested? Updated the baseline of the pretrain_minimax_m2p7_4K_fsdp2 test case. This test case was tested in the CI gate. See merge request: Ascend/MindSpeed-LLM!4670 | 2 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 6 个月前 | ||
| 2 个月前 | ||
| 25 天前 | ||
| 6 个月前 | ||
| 1 个月前 | ||
| 2 个月前 | ||
| 1 个月前 | ||
| 6 个月前 | ||
| 3 个月前 | ||
| 2 个月前 |