| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix: enable train_from_hf for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4642 merge Train_from_HF into master fix: enable train_from_hf for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? MindSpeed-LLM now supports online loading of datasets and model weights during training, but the legacy scripts in /examples/mcore have not yet been enabled to use this new feature. This PR enables these pretraining and fine-tuning scripts to use online data and weight loading. reference: [https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md) ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4642 | 2 个月前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
fix: enable train_from_hf for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4642 merge Train_from_HF into master fix: enable train_from_hf for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? MindSpeed-LLM now supports online loading of datasets and model weights during training, but the legacy scripts in /examples/mcore have not yet been enabled to use this new feature. This PR enables these pretraining and fine-tuning scripts to use online data and weight loading. reference: [https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md) ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4642 | 2 个月前 | |
fix(PyTorch): fix hf2mg of ds4 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4956 merge master into master fix(PyTorch): fix hf2mg of ds4 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR updates the Hugging Face (HF) to Megatron (MG) weight conversion workflow for the DeepSeek-V4 (DS4) model. The main change is to make the existing DS4 weight conversion process compatible with the DS4-0731 model version. The conversion logic has been adjusted accordingly so that DS4-0731 checkpoints can be converted correctly between HF and MG formats. Currently, the DS4-0731 model with MTP enabled is not supported by this conversion workflow. ## Does this PR introduce any user-facing change? Yes. Users can now use the HF-MG weight conversion workflow with the DS4-0731 model. There are no additional usage changes for other DS4 model versions. Note that DS4-0731 checkpoints with the MTP parameter enabled are not supported. ## How was this patch tested? The patch was tested by running the HF-to-MG and MG-to-HF weight conversion workflow with the DS4-0731 model and verifying that the conversion completes successfully. Usage constraint: * DS4-0731 is supported when MTP is disabled. * DS4-0731 with MTP enabled is currently not supported. See merge request: Ascend/MindSpeed-LLM!4956 | 16 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat(mcore): add GLM-5.2 A5 POC scripts Co-authored-by: LinShua<707894133@qq.com> # message auto-generated for no-merge-commit merge: !5013 merge master_glm52_A5_sh into master feat(mcore): add GLM-5.2 A5 POC scripts Created-by: LinShua Commit-by: LinShua Merged-by: ascend-robot Description: Fixes #1825 ## What this PR does / why we need it? 1. Replaces the existing GLM-5.2 32B A5 POC entry with a 51B 4K FP8 configuration using TP1/PP2/EP4 parallelism, Transformer Engine MXFP8, index-topk=1024, fixed routing, the pipeline layout, and a matching log name. 2. Adds a GLM-5.2 210B 4K A5 FP8 POC entry with an eight-node TP1/PP2/EP32 configuration, Transformer Engine MXFP8 options, and portable data/tokenizer/checkpoint path placeholders. 3. Aligns --lr-warmup-iters to 500 in the GLM-5.2 744B A3 example, 744B A3 POC, and 210B A3 POC scripts. 4. This change is limited to five shell launch configurations and does not modify training runtime code or public APIs. ## Does this PR introduce any user-facing change? Yes. Users can run the new tests/poc/glm52/pretrain_glm52_210b_4k_A5_fp8_ptd.sh entry and the renamed tests/poc/glm52/pretrain_glm52_51b_4k_A5_ptd.sh entry. The 51B entry now uses the A5 MXFP8 and TP1/PP2/EP4 configuration, and the related A3 scripts now default to 500 learning-rate warmup iterations. ## How was this patch tested? 1. bash -n passed for all five changed shell scripts. 2. Changed-file pre-commit passed trailing-whitespace, end-of-file, added-large-file, merge-conflict, private-key, codespell, and typos checks. YAML, JSON, Python, and C/C++ hooks had no matching files. The repository-local gitleaks-offline-scan hook could not run because the required ./gitleaks executable is not present in the repository or local environment. 3. git diff --cached --check passed before amend. 4. The remote Git hook passed during git push --force-with-lease. 5. Functional, accuracy, performance, and end-to-end NPU training tests were not run. See merge request: Ascend/MindSpeed-LLM!5013 | 2 天前 | |
feat(pytorch): add hy3 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4781 merge master into master feat(pytorch): add hy3 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? - Add Hunyuan3 model adaptation. - Add Hunyuan3 model configuration. - Add Hunyuan3 weight conversion support. - Integrate Hunyuan3 into the existing model loading and training workflow. ## Does this PR introduce any user-facing change? Yes. This PR introduces support for the Hunyuan3 model. Users can now use MindSpeed-LLM to perform weight conversion and training for Hunyuan3 following the existing workflow. No user-facing changes are introduced for existing supported models. ## How was this patch tested? The following tests have been completed: - Pipeline ST passed. - Verified that Hunyuan3 weight conversion completes successfully. - Verified that the converted checkpoint can be loaded successfully. - Verified that the training workflow starts successfully after conversion. See merge request: Ascend/MindSpeed-LLM!4781 | 1 个月前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
fix: enable train_from_hf for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4642 merge Train_from_HF into master fix: enable train_from_hf for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? MindSpeed-LLM now supports online loading of datasets and model weights during training, but the legacy scripts in /examples/mcore have not yet been enabled to use this new feature. This PR enables these pretraining and fine-tuning scripts to use online data and weight loading. reference: [https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md](https://gitcode.com/Ascend/MindSpeed-LLM/blob/master/docs/zh/pytorch/training/pretrain/mcore/train_from_hf.md) ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4642 | 2 个月前 | |
fix: fix parameter name typo Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4953 merge Train_from_HF into master fix: fix train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Fixes a parameter name typo by changing --tokenizer-type PretrainedFormHF to --tokenizer-type PretrainedFromHF. Kimi2 does not currently support online weight conversion, removes the default enablement of online weight conversion in the script. ## Does this PR introduce any user-facing change? No. ## How was this patch tested? Verified through CI testing. See merge request: Ascend/MindSpeed-LLM!4953 | 19 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
refactor(pytorch): delete ckpt-v1 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4831 merge master into master refactor(pytorch): delete ckpt-v1 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR removes the legacy Weight Conversion V1 implementation and completes the migration to Weight Conversion V2. The main changes include: * Remove the deprecated Weight Conversion V1 codebase. * Migrate the remaining scripts that still depended on V1 to Weight Conversion V2. * Update related README and documentation to reflect the new conversion workflow. * Clean up obsolete files and references associated with the V1 implementation. This change simplifies code maintenance and ensures that all weight conversion workflows are unified under the actively maintained V2 framework. --- ## Does this PR introduce any user-facing change? Yes. Weight Conversion V1 is no longer supported. Users should use Weight Conversion V2 for all checkpoint conversion workflows. The related README and documentation have been updated accordingly to guide users to the new conversion process. --- ## How was this patch tested? The following verification was performed: * Verified that all remaining weight conversion scripts have been successfully migrated to Weight Conversion V2. * Verified that the updated conversion workflow functions correctly after removing the V1 implementation. * Verified that related documentation and examples are consistent with the current implementation. * Regression tests and pipeline ST passed successfully. See merge request: Ascend/MindSpeed-LLM!4831 | 1 个月前 | |
feat(pytorch): add model minimax27 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !4735 merge master into master feat(pytorch): add model minimax27 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds support for the MiniMax-M27 model in MindSpeed-LLM, enabling users to perform model conversion and training with the MiniMax-M27 architecture. - Add MiniMax-M27 model adaptation. - Add the corresponding model configuration and weight conversion support. - Enable the MiniMax-M27 workflow in the existing training/conversion pipeline. ## Does this PR introduce any user-facing change? Yes. This PR introduces support for the MiniMax-M27 model. Users can now use MindSpeed-LLM to perform weight conversion and training for MiniMax-M27 following the existing workflow. No changes are required for existing models. ## How was this patch tested? The following tests have been completed: - Pipeline ST passed. - Verified that MiniMax-M27 weight conversion completes successfully. - Verified that the converted model can be loaded correctly. - Verified that the training workflow starts successfully after conversion. See merge request: Ascend/MindSpeed-LLM!4735 | 1 个月前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 21 天前 |
| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
| 2 个月前 | ||
| 21 天前 | ||
| 2 个月前 | ||
| 16 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 2 天前 | ||
| 1 个月前 | ||
| 21 天前 | ||
| 2 个月前 | ||
| 19 天前 | ||
| 21 天前 | ||
| 1 个月前 | ||
| 1 个月前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 | ||
| 21 天前 |