master

ascend-robotfeat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs

6d9dd076创建于 17 天前历史提交

文件	最后提交记录	最后更新时间
pretrain_mamba3_demo_1b_2K_fsdp2_A2.sh	feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742	17 天前
pretrain_mamba3_demo_1b_2K_fsdp2_A2.yaml	feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !4742 merge master into master feat: move data.dataset from YAML to shell script CLI args across all FSDP2 training configs Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: [#1631](https://gitcode.com/Ascend/MindSpeed-LLM/issues/1631) ## What this PR does / why we need it? Moves the data.dataset field from YAML configs to shell script CLI arguments across all FSDP2 training scripts, ensuring consistency with the existing pattern where environment-specific parameters (paths, parallel sizes, batch sizes) are passed via CLI. Adds deep-merge support in argument parsing so that when a dict-type dataset field is provided via CLI, its keys are merged with (rather than replacing) any YAML defaults. ## Does this PR introduce any user-facing change? Yes. Users must now provide --data.dataset via CLI ( .sh ) instead of YAML. For dict-type datasets, the value is a JSON string (e.g., --data.dataset '{"file_name": "..."}' ). For string-type datasets, it's passed as-is (e.g., --data.dataset alpaca_full ). CLI dict keys take precedence over YAML dict keys via deep merge. Affected files: all examples/fsdp2/**/*.sh|.yaml and tests/**/*fsdp2*.sh|.yaml . ## How was this patch tested? Verified argument parsing logic with unit-level code review; confirmed CLI --data.dataset correctly overrides/merges with YAML. Existing pre-commit hooks (ruff, pylint, yaml check) passed. Full training run recommended to validate end-to-end data loading. See merge request: Ascend/MindSpeed-LLM!4742	17 天前