| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
docs: fix text pretraining configuration guide Co-authored-by: yeqm<yeqiangmao@huawei.com> # message auto-generated for no-merge-commit merge: !2974 merge docs/fix-text-pretrain-guide into master docs: fix text pretraining configuration guide Created-by: ye_qm Commit-by: yeqm Merged-by: ascend-robot Description: ## What this PR does / why we need it? The FSDP2 text pretraining example in docs/zh/features/pretrain.md does not match the current implementation. The existing example only sets attr.pretrain: true. However, the FSDP2 backend selects PretrainDatasetProcessor through basic_parameters.stage. Without stage: pretrain, the stage defaults to sft, causing text-only samples without a response field to be dropped and eventually resulting in StopIteration. This PR: - Adds the required basic_parameters.stage: pretrain setting for the FSDP2 backend. - Replaces the deprecated collate_param.model_name with collate_param.collator_id. - Clarifies the configuration differences between the FSDP2 and Megatron backends. - Corrects the packing description: text is packed within each preprocessing batch, so each batch must contain at least cutoff_len valid tokens to generate a training sample. - Improves the formatting and readability of the guide. Related issue: N/A. The problem was found while verifying the documented workflow against the current master implementation. ## Does this PR introduce any user-facing change? Yes. This PR corrects the user-facing configuration required to run text-only pretraining with the FSDP2 backend. Users should configure: ```yaml data: dataset_param: attr: formatting: alpaca prompt: text basic_parameters: stage: pretrain template: default dataloader_param: collate_param: collator_id: llm_pretrain See merge request: Ascend/MindSpeed-MM!2974 | 1 个月前 | |
docs: AIDD扫描问题处理 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2849 merge 0715_AIDD into master docs: AIDD扫描问题处理 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? AIDD扫描问题处理 ## Does this PR introduce any user-facing change? AIDD扫描问题处理 ## How was this patch tested? readthedoc See merge request: Ascend/MindSpeed-MM!2849 | 2 个月前 | |
docs:update supported_models Co-authored-by: z60112595<zhaotao68@h-partners.com> # message auto-generated for no-merge-commit merge: !2872 merge master into master docs:update supported_models Created-by: kali00 Commit-by: z60112595 Merged-by: ascend-robot Description: ## What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed-MM/issues/517 更新支持的模型列表，删除下架模型 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2872 | 2 个月前 | |
docs:rename Ascend Extension for PyTorch Co-authored-by: z60112595<zhaotao68@h-partners.com> # message auto-generated for no-merge-commit merge: !2836 merge master into master docs:rename Ascend Extension for PyTorch Created-by: kali00 Commit-by: z60112595 Merged-by: ascend-robot Description: ## What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed-MM/issues/506 1、更新Ascend Extension for PyTorch名称为TorchNPU 2、补充快速入门环境最低配置 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2836 | 2 个月前 | |
docs: AIDD扫描问题处理 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2849 merge 0715_AIDD into master docs: AIDD扫描问题处理 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? AIDD扫描问题处理 ## Does this PR introduce any user-facing change? AIDD扫描问题处理 ## How was this patch tested? readthedoc See merge request: Ascend/MindSpeed-MM!2849 | 2 个月前 | |
docs: AIDD扫描问题处理 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2849 merge 0715_AIDD into master docs: AIDD扫描问题处理 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? AIDD扫描问题处理 ## Does this PR introduce any user-facing change? AIDD扫描问题处理 ## How was this patch tested? readthedoc See merge request: Ascend/MindSpeed-MM!2849 | 2 个月前 | |
docs: 移除文档中 MindSpore 相关展示 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2670 merge docs-fix into master docs: 移除文档中 MindSpore 相关展示 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? 移除文档中 MindSpore 相关展示(目录树/Profiler项),仅保留 PyTorch 移除出海描述 ## Does this PR introduce any user-facing change? 移除文档中 MindSpore 相关展示(目录树/Profiler项),仅保留 PyTorch 移除出海描述 ## How was this patch tested? read See merge request: Ascend/MindSpeed-MM!2670 | 3 个月前 | |
docs:rename Ascend Extension for PyTorch Co-authored-by: z60112595<zhaotao68@h-partners.com> # message auto-generated for no-merge-commit merge: !2836 merge master into master docs:rename Ascend Extension for PyTorch Created-by: kali00 Commit-by: z60112595 Merged-by: ascend-robot Description: ## What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed-MM/issues/506 1、更新Ascend Extension for PyTorch名称为TorchNPU 2、补充快速入门环境最低配置 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2836 | 2 个月前 | |
modify: remove dependency on third-party libraries Co-authored-by: WendongPang<pangwendong@huawei.com> # message auto-generated for no-merge-commit merge: !2719 merge remove into master modify: remove dependency on third-party libraries Created-by: WendongPang Commit-by: WendongPang Merged-by: ascend-robot Description: ## What this PR does / why we need it? modify: remove dependency on third-party libraries https://gitcode.com/Ascend/MindSpeed-MM/issues/311 ## Does this PR introduce any user-facing change? 开放了pandas, accelerate, peft的版本依赖，其中peft完成了megatron后端lora功能的适配，兼容旧版本0.7.1和当前最新版本0.19.1，移除了timm及相关的模型 ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2719 | 3 个月前 | |
docs:rename Ascend Extension for PyTorch Co-authored-by: z60112595<zhaotao68@h-partners.com> # message auto-generated for no-merge-commit merge: !2836 merge master into master docs:rename Ascend Extension for PyTorch Created-by: kali00 Commit-by: z60112595 Merged-by: ascend-robot Description: ## What this PR does / why we need it? https://gitcode.com/Ascend/MindSpeed-MM/issues/506 1、更新Ascend Extension for PyTorch名称为TorchNPU 2、补充快速入门环境最低配置 ## Does this PR introduce any user-facing change? Please describe whether the PR will result in any user-facing usage changes. If there is related documentation, please specify its path. ## How was this patch tested? Please explain how to verify the correctness and effectiveness of this feature, as well as its usage constraints and limitations. See merge request: Ascend/MindSpeed-MM!2836 | 2 个月前 | |
修改路径替换脚本默认路径, docker描述修改 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2736 merge 0630_npupath into master feat: 修改路径替换脚本默认路径修改, 修改docker描述 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? 按新要求,修改默认路径.验证docker, 修改docker场景下使用描述 ## Does this PR introduce any user-facing change? 按新要求,修改默认路径.验证docker, 修改docker场景下使用描述 ## How was this patch tested? 测试该使用场景下, 模型正常训练 See merge request: Ascend/MindSpeed-MM!2736 | 3 个月前 | |
docs:fsdp2 特性文档后端说明补全 Co-authored-by: liuxi_<liuxi75@huawei.com> # message auto-generated for no-merge-commit merge: !2702 merge fix_doc_0616 into master docs:fsdp2 特性文档后端说明补全 Created-by: liuxi_ Commit-by: liuxi_ Merged-by: ascend-robot Description: ## What this PR does / why we need it? 对supported_features.md进行排查,确保描述的特性与支持后端相符. 对fsdp2后端的特性文档补全原生fsdp2的相关说明(目前以megatron-fsdp为主) ## Does this PR introduce any user-facing change? fsdp2 特性文档后端说明补全 ## How was this patch tested? read and read See merge request: Ascend/MindSpeed-MM!2702 | 3 个月前 | |
feat(fsdp): add autograd anomaly detection Co-authored-by: ye_qm<yeqiangmao@huawei.com> # message auto-generated for no-merge-commit merge: !2966 merge yqm/autograd-anomaly-detection into master feat(fsdp): add autograd anomaly detection Created-by: ye_qm Commit-by: ye_qm Merged-by: ascend-robot Description: `` markdown ## What this PR does / why we need it? PyTorch autograd errors, such as backward operator failures and NaN gradients, can be difficult to locate because the error may not identify the corresponding forward operation. This PR adds environment-variable-controlled autograd anomaly detection for the MindSpeed-MM FSDP2 training path. Detailed changes: - Register a new FSDP2 environment variable: - Name: MM_DETECT_ANOMALY - Type: bool - Default: false - Read the environment variable through the centralized FSDP2 environment-variable manager. - Configure PyTorch anomaly detection during trainer initialization: python detect_anomaly = envs.MM_DETECT_ANOMALY torch.autograd.set_detect_anomaly(detect_anomaly) - Enable anomaly detection before model execution so backward failures can include the corresponding forward traceback. - Print a warning when anomaly detection is enabled because it significantly reduces training performance. - Apply the setting through the common Trainer.initialize() path, covering both the standard FSDP2 trainer and the FunASR trainer. - Add tests for: - the default disabled state; - boolean environment-variable conversion; - propagation of the environment-variable value to PyTorch. - Add troubleshooting documentation describing how to enable the feature and its performance limitations. - Replace the direct LOCAL_RANK environment read in model_container_hub.py with envs.LOCAL_RANK, keeping FSDP2 business code consistent with centralized environment-variable management. This is a debugging feature rather than a bugfix, so there is no related issue. ## Does this PR introduce any user-facing change? Yes. Users can enable autograd anomaly detection through the following environment variable: bash export MM_DETECT_ANOMALY=true The environment variable supports the project-wide boolean formats: - True values: 1, true, yes, y, on - False values: 0, false, no, n, off Values are case-insensitive and may contain surrounding whitespace. If MM_DETECT_ANOMALY is unset, it defaults to false. Therefore, existing training scripts and default runtime behavior are unchanged. To disable it explicitly: bash export MM_DETECT_ANOMALY=false Related documentation: text docs/zh/troubleshooting.md ## How was this patch tested? The following checks were performed: - Environment-variable unit tests: bash pytest tests/ut_fsdp/test_envs.py -q Result: text 63 passed - Python syntax compilation for all modified Python files. - git diff --check to detect whitespace errors. - Ruff and Black checks for the modified test files. - GitCode remote Git Hooks checks passed. The following test files cover this feature: text tests/ut_fsdp/test_envs.py tests/ut_fsdp/train/test_autograd_anomaly.py The tests verify that: 1. MM_DETECT_ANOMALY defaults to false when it is not exported. 2. Supported true and false strings are converted to Python bool values. 3. Trainer.initialize() calls: python torch.autograd.set_detect_anomaly(False) when the environment variable is disabled. 4. Trainer.initialize() calls: python torch.autograd.set_detect_anomaly(True) when the environment variable is enabled. 5. FSDP2 business code does not introduce unapproved direct os.getenv() or os.environ.get() calls. The focused trainer test can be run with: bash pytest tests/ut_fsdp/train/test_autograd_anomaly.py -v Usage constraints and limitations: - This feature is intended only for debugging. - Enabling it significantly reduces training performance. - It should remain disabled during formal or performance-sensitive training. - It helps trace backward failures to their corresponding forward operations, but it does not guarantee detection of every numerical issue. - In distributed training, anomaly information is reported by the rank on which the failure occurs. - The local development environment did not contain PyTorch, so the PyTorch-dependent trainer test was added but could not be executed locally. It should be executed by the project CI environment. `` See merge request: Ascend/MindSpeed-MM!2966 | 1 个月前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 1 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 2 个月前 | ||
| 3 个月前 | ||
| 3 个月前 | ||
| 1 个月前 |