| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 | |
feat: extend train_from_hf support for legacy scripts Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4880 merge Train_from_HF into master feat: extend train_from_hf support for legacy scripts Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Enabled automatic data preprocessing for all training scripts under examples and tests/poc. Enabled automatic weight conversion for training scripts of all models below 300B parameters under examples and tests/poc (excluding DeepSeek3, DeepSeek4, and LongCat). ## Does this PR introduce any user-facing change? The updated scripts integrate data preprocessing, weight conversion, and training into a single workflow, allowing users to launch training jobs with a single script. The default --tokenizer-type in the Llama2 training script has been changed from Llama2Tokenizer to PretrainedFormHF, and --tokenizer-model ${TOKENIZER_MODEL} has been changed to --tokenizer-name-or-path ${TOKENIZER_MODEL}. TOKENIZER_MODEL only needs to be configured to the directory containing the target model's tokenizer. ## How was this patch tested? This PR was validated through smoke testing. See merge request: Ascend/MindSpeed-LLM!4880 | 24 天前 |