| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
feat: refactor zero-experts MoE dispatcher pipeline and adapt to 018 signatures Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !5133 merge master into master feat: refactor zero-experts MoE dispatcher pipeline and adapt to 018 signatures Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adapts LongCat-Flash to Megatron-Core 0.18 and restores numeric parity with the 0.12 baseline: LinearNoTP is rebased on TE Linear for native FP8, zero-experts is rewritten on the 0.18 staged token dispatcher, MLA softmax scale are pinned to the 0.12 semantics, 0.18 signatures (pg_collection/vp_stage, generate_state_dict) are aligned, the SafeUnpickler allowlist is extended for TE-NPU recipes. Without these changes LongCat fails to build or load/save checkpoints on 0.18, and training diverges numerically from the 0.12 baseline. (Internal 012/018 alignment analysis; no public issue.) ## Does this PR introduce any user-facing change? No change to training results or model behavior. The only user-visible difference is that all longcat launch scripts now explicitly pass --no-moe-npu-fast-ops (harmless, as fast ops is conflict-disabled anyway when --num-zero-experts is set). No additional documentation is required. ## How was this patch tested? Verified on an 8-card Ascend A5 environment with pretrain_longcat_flash_9.7b_8k_fp8_A5_ptd.sh: a 50-iteration resume from the 0.12 iteration-10 checkpoint reproduces the 0.12 baseline with bitwise-identical loss/grad norm (token streams sha1-identical), and checkpoints are saved and reloaded correctly. Constraints: keep --ckpt-format torch_dist when loading the 0.12 init checkpoint, and longcat runs enabling --num-zero-experts must keep --no-moe-npu-fast-ops unless the fast_ops_conflicts defense in MindSpeed is present. See merge request: Ascend/MindSpeed-LLM!5133 | 18 天前 | |
fix(pytorch):fix 018 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !5153 merge master into master fix(pytorch):fix 018 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? Register the required arguments and adapt to a function interface change to maintain compatibility. ## Does this PR introduce any user-facing change? No changes to existing usage. ## How was this patch tested? Verify argument parsing and run the affected code path to confirm compatibility with the updated function interface. See merge request: Ascend/MindSpeed-LLM!5153 | 18 天前 | |
fix: guard indexer loss scaler against zero num_items_in_batch Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !5083 merge guard_indexer_loss into master fix: guard indexer loss scaler against zero num_items_in_batch Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: ### What this PR does / why we need it? Fixes a ZeroDivisionError in IndexerLossAutoScaler.set_loss_scale_from_batch_stats: when num_items_in_batch is 0 (e.g. every label in the gradient-accumulation window is ignore_index on all ranks), the scaler now falls back to 1 / gradient_accumulation_steps instead of crashing on 1.0 / float(0). ### Does this PR introduce any user-facing change? No change for normal training; only the previously-fatal zero-valid-token batch now trains on with the accum-steps fallback scale, and no documentation update is required. ### How was this patch tested? Verified by invoking set_loss_scale_from_batch_stats with (0, N) and (None, N), confirming both now apply the 1 / N accum-steps fallback while positive token counts still yield 1 / num_items_in_batch; the normal path remains covered by the existing fsdp2 trainer tests. See merge request: Ascend/MindSpeed-LLM!5083 | 1 个月前 | |
Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 1 个月前 | |
chore: Remove redundancy and refactor Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !5074 merge redundancy0918 into master chore: Remove redundancy and refactor Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: **What this PR does / why we need it?** This PR cleans up dead code and unused features under tests/mindspeed_llm, and removes the temporary _v2 naming from the adaptor layer: 1. **Rename temporary v2 symbols back to canonical names** - mindspeed_llm.tasks.megatron_adaptor_v2 → megatron_adaptor - training.arguments.process_args_v2 → process_args - validate_args_v2_decorator → validate_args_decorator 2. **Remove use_fused_mlp fused SwiGLU MLP path** (never registered in the feature list) - Delete tasks/models/transformer/fast_mlp.py (_FusedParallelMLPFunction, ParallelSwigluMLPForward) - Delete tasks/posttrain/lora/cc_lora_mlp_forward.py (_FusedBaseMLPLoRAFunction, LoRA fused MLP forward) - Remove the args.use_fused_mlp monkey-patch branches in training/training.py 3. **Remove shared-memory data transfer path** (--enable-share-memory) - Delete tasks/dataset/shared_memory_manager.py (SharedMemoryManager) - Remove the global shm manager (get_sharedmem_mgr / reset_sharedmem_mgr / shm branch of get_batch_on_this_tp_rank) in training/utils.py - Delete the DataFeature feature class (features_manager/common/data.py) 4. **Remove return_document_ids support** - Remove gpt_dataset_getitem_wrapper from core/datasets/gpt_dataset.py and its patch registration in DatasetFeature - Remove per-iteration document_ids debug printing in DecoderPackedMTFDataset 5. **Remove pairwise-dataset support** (is_pairwise_dataset) - Delete tasks/preprocess/collator.py (PairwiseDataCollatorWithPadding) - Drop pairwise branches in legacy/data/data_samplers.py, tasks/posttrain/utils.py, and DecoderPackedMTFDataset 6. **Drop the _add_dummy_args_v2 compatibility shim** - No longer pre-seed unsupported args (unaligned_linear, embed_layernorm, enable_share_memory, return_document_ids, attention_mask_on_cpu, output_layer_slice_num, use_fused_mlp, is_pairwise_dataset); remove the unused UnalignedLinearFeature and TrainingDefaultFeature accordingly. **Does this PR introduce any user-facing change?** No. All removed paths were dead code: the deleted features were never exposed via the feature manager list (only seeded by the dummy-args shim), and the renamed functions are internal symbols. Training/post-training behavior is unchanged. **How was this patch tested?** Locally verified import mindspeed_llm and args-parsing paths still resolve after the renames; existing CI pipeline passes (docs-ci-pipeline-success). See merge request: Ascend/MindSpeed-LLM!5074 | 1 个月前 | |
fix: prevent TND CP hang on empty compressed-KV ranks Co-authored-by: ByteDance<lihaozhe9@huawei.com> # message auto-generated for no-merge-commit merge: !5086 merge fix_hang_v5 into master fix: prevent TND CP hang on empty compressed-KV ranks Created-by: lihaozhe777 Commit-by: ByteDance Merged-by: ascend-robot Description: ### What this PR does / why we need it? This PR fixes a backward hang in DeepSeek-V4 CSA training with packed TND sequences and Context Parallelism. A CP rank may contain tokens but have no document segment long enough to produce a complete compressed-KV block. Previously, empty ranks either skipped parts of the indexer/compressor graph or used dummy KV, top-k indices, and weights to enter the fused path. This could cause inconsistent autograd graphs, parameter hooks, gradient buckets, and collective ordering across CP ranks. The fused indexer-loss backward also performs a CP all_gather . When only non-empty ranks entered this collective, training could hang. This patch: - Detects ranks without valid compressed KV before indexer execution. - Falls back to sliding-window attention on empty ranks. - Removes dummy compressed KV, top-k indices, and weights. - Adds zero-gradient anchors for indexer and compressor parameters to preserve gradient hooks and bucket ordering. - Adds a backward communication anchor so empty ranks join the fused indexer-loss CP collective. - Records zero indexer loss on empty ranks to keep loss tracker keys aligned. - Extracts the shared TND token-count collective into gather_tnd_local_num() . - Updates only the required tests/mindspeed_llm CI mirror files. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? - Ran Python syntax checks with py_compile . - Ran git diff --check . - Ran repository pre-commit checks, including Ruff, Pylint, Bandit, typos, and secret scanning. - Verified that only the required DeepSeek-V4 CI mirror files were changed. - Reviewed both empty-rank and non-empty-rank control flow for TND packed sequences. See merge request: Ascend/MindSpeed-LLM!5086 | 1 个月前 | |
fix(pytorch):fix 018 Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !5153 merge master into master fix(pytorch):fix 018 Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? Register the required arguments and adapt to a function interface change to maintain compatibility. ## Does this PR introduce any user-facing change? No changes to existing usage. ## How was this patch tested? Verify argument parsing and run the affected code path to confirm compatibility with the updated function interface. See merge request: Ascend/MindSpeed-LLM!5153 | 18 天前 | |
feat(pytorch): 018 upgrade CP adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5096 merge CP into master feat(pytorch): 018 upgrade CP adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation of the CP features ulysses_cp, megatron_cp, and kvallgather_cp to upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy alignment between 018 with CP=1, ulysses_cp_algo, kvallgather_cp_algo, and megatron_cp_algo and 012 with CP=1. Performance and memory usage show no degradation using relative comparison methods. See merge request: Ascend/MindSpeed-LLM!5096 | 1 个月前 | |
feat(pytroch): Merge upgrade 018 base code Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !4999 merge prepare_018 into master feat(pytroch): Merge upgrade 018 base code Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR serves as the foundational update for 018 adaptation. It adds the mindspeed_llm directory under ests and removes DPOFeature, MambaModel, HighAvailabilityFeature, ai_framework_feature, and layerwise_disaggregated_training_feature to facilitate the subsequent integration of code related to the 018 upgrade. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? This PR only merges the basic code and does not go through testing; it will be verified during subsequent adaptation. See merge request: Ascend/MindSpeed-LLM!4999 | 2 个月前 | |
feat: refactor zero-experts MoE dispatcher pipeline and adapt to 018 signatures Co-authored-by: Rostellaria<xiaoshaoan@huawei.com> # message auto-generated for no-merge-commit merge: !5133 merge master into master feat: refactor zero-experts MoE dispatcher pipeline and adapt to 018 signatures Created-by: Rostellaria Commit-by: Rostellaria Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adapts LongCat-Flash to Megatron-Core 0.18 and restores numeric parity with the 0.12 baseline: LinearNoTP is rebased on TE Linear for native FP8, zero-experts is rewritten on the 0.18 staged token dispatcher, MLA softmax scale are pinned to the 0.12 semantics, 0.18 signatures (pg_collection/vp_stage, generate_state_dict) are aligned, the SafeUnpickler allowlist is extended for TE-NPU recipes. Without these changes LongCat fails to build or load/save checkpoints on 0.18, and training diverges numerically from the 0.12 baseline. (Internal 012/018 alignment analysis; no public issue.) ## Does this PR introduce any user-facing change? No change to training results or model behavior. The only user-visible difference is that all longcat launch scripts now explicitly pass --no-moe-npu-fast-ops (harmless, as fast ops is conflict-disabled anyway when --num-zero-experts is set). No additional documentation is required. ## How was this patch tested? Verified on an 8-card Ascend A5 environment with pretrain_longcat_flash_9.7b_8k_fp8_A5_ptd.sh: a 50-iteration resume from the 0.12 iteration-10 checkpoint reproduces the 0.12 baseline with bitwise-identical loss/grad norm (token streams sha1-identical), and checkpoints are saved and reloaded correctly. Constraints: keep --ckpt-format torch_dist when loading the 0.12 init checkpoint, and longcat runs enabling --num-zero-experts must keep --no-moe-npu-fast-ops unless the fast_ops_conflicts defense in MindSpeed is present. See merge request: Ascend/MindSpeed-LLM!5133 | 18 天前 | |
chore: Rename _IS_018 to _VERSION_018 Co-authored-by: iansheng<shengjiayi@huawei.com> # message auto-generated for no-merge-commit merge: !5076 merge redundancy0918 into master chore: Rename _IS_018 to _VERSION_018 Created-by: iansheng Commit-by: iansheng Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR continues the cleanup follow-up of the previous redundancy refactor: 1. Rename the version flag _IS_018 to _VERSION_018 in convert_ckpt_v2.py and pretrain_gpt.py for clarity. 2. Remove the redundant as megatron_adaptor self-alias in tests/mindspeed_llm/__init__.py. ## Does this PR introduce any user-facing change? No. These are internal naming and import cleanups; behavior is unchanged. ## How was this patch tested? N/A See merge request: Ascend/MindSpeed-LLM!5076 | 1 个月前 |