| Files | Last commit | Last update |
|---|---|---|
doc: fix dataset paths in the pretraining docs Co-authored-by: daixzh<daixuezheng@huawei.com> # message auto-generated for no-merge-commit merge: !4718 merge pretrain_doc_fix into master doc: fix dataset paths in the pretraining docs Created-by: daixzh Commit-by: daixzh Merged-by: ascend-robot Description: ## What this PR does / why we need it? Fixes incorrect dataset paths in the pretraining documentation. ## Does this PR introduce any user-facing change? No. ## How was this patch tested? This change only updates the documentation and has passed the docs-ci. See merge request: Ascend/MindSpeed-LLM!4718 | 4 days ago | |
feat: add MTP loss support with mtp_loss_scaling_factor parameter Co-authored-by: wxy1105<wangxinyu136@huawei.com> # message auto-generated for no-merge-commit merge: !4633 merge master_train into master feat: add MTP loss support with mtp_loss_scaling_factor parameter Created-by: wxy1105 Commit-by: wxy1105 Merged-by: ascend-robot Description: ## What this PR does / why we need it? Background : This PR adds MTP (Multi Token Prediction) loss support for DeepSeek-V4 model training in FSDP2 backend, enabling multi-step prediction during training to improve model performance. Detailed changes : -Added mtp_loss_scaling_factor parameter ( default=0.3 ) in arguments.py for controlling MTP loss weight -Created LossOutput dataclass in trainer.py to encapsulate total loss, LM loss, and MTP loss -Modified _compute_loss method to calculate and accumulate MTP loss from model outputs -Updated training_step to return separate LM loss and MTP loss values for logging -Enhanced train_monitor.py to log MTP loss metrics alongside LM loss -Added documentation for mtp_loss_scaling_factor in arguments.md ## Does this PR introduce any user-facing change? Yes : -New mtp_loss_scaling_factor parameter available in FSDP2 training configuration -Training logs now include separate lm loss and mtp loss metrics -Related docs: arguments.md ## How was this patch tested? -Compilation and unit tests for FSDP2 trainer with MTP loss integration -Training monitor logging validation with MTP metrics -End-to-end training tests with DeepSeek-V4 models See merge request: Ascend/MindSpeed-LLM!4633 | 4 days ago |