| 文件 | 最后提交记录 | 最后更新时间 |
|---|---|---|
fix(pytorch): add copyright Co-authored-by: qyzqyz<quyueze@h-partners.com> # message auto-generated for no-merge-commit merge: !5068 merge master into master fix(pytorch): add copyright Created-by: qyzqyz Commit-by: qyzqyz Merged-by: ascend-robot Description: ## What this PR does / why we need it? add copyright from nvidia ## Does this PR introduce any user-facing change? no ## How was this patch tested? just add copyright of nvidia before code See merge request: Ascend/MindSpeed-LLM!5068 | 30 天前 | |
fix:Tried to allocate more than 1EB memory in deepseekv4 training Co-authored-by: jy<jiayuan22@huawei.com> # message auto-generated for no-merge-commit merge: !5101 merge feature into master fix:Tried to allocate more than 1EB memory in deepseekv4 training Created-by: gcw_m1ionjwU Commit-by: jy Merged-by: ascend-robot Description: Problem Statement / Root Cause In Pipeline Parallelism (PP) scenarios involving non-uniform activation shapes (e.g., DeepSeek-V4 with MHC enabled or coexisting 3D/4D tensors), VERL's Megatron static patch hardcodes the receive shape buffer size to 4. When the sender transmits a 3D tensor (3 values), the receiver still expects 4 values, reading uninitialized memory garbage for the 4th dimension. This causes torch.empty() to attempt allocating massive memory (>1 EB), leading to a pseudo-OOM crash. Additionally, MindSpeed-LLM fails to fallback to the dynamic shape communication logic due to two integration issues: * variable_seq_lengths parameter timing issue during the patching lifecycle. * Missing VariableSequenceLengthFeature registration in MindSpeed-LLM's FEATURES_LIST. Key Changes * Feature Registration: Explicitly added VariableSequenceLengthFeature to MindSpeed-LLM's feature patch list to enable dynamic rank-aware shape exchange (3D vs 4D). * Configuration Propagations: Enabled passing variable_seq_lengths=True via runner configurations to ensure proper parameter penetration before the patching phase. issue:https://gitcode.com/Ascend/MindSpeed-LLM/issues/1897 See merge request: Ascend/MindSpeed-LLM!5101 | 24 天前 | |
feat(fsdp2): add Qwen3-4B A3 pretraining support Co-authored-by: hahahascend<serverlessllm@163.com> # message auto-generated for no-merge-commit merge: !4966 merge codex/issue1353-qwen3-4b-fsdp2 into master feat(fsdp2): add Qwen3-4B A3 pretraining support Created-by: hahahascend Commit-by: hahahascend Merged-by: ascend-robot Description: Fixes #1353 ## What this PR does / why we need it? This PR adds Qwen3-4B pretraining support to the FSDP2 backend on 4 Ascend NPUs with a 4K sequence length. - Adds a Qwen3-4B FSDP2 launch script and YAML configuration with fused RMSNorm, fused rotary position embedding, Flash Attention, and parameter resharding after forward. - Loads mx_quant_group_gemm only when MXFP8 MoE quantization is enabled. - Adds the Ascend910_93 short SoC prefix with a BF16 peak performance of 354 TFLOPS for MFU calculation on A3 devices such as Ascend910_9382. - Enables reshard_after_forward=True by default to provide stable iteration time and memory usage. ## Does this PR introduce any user-facing change? Yes. Qwen3-4B FSDP2 pretraining can be launched with: bash MODEL_PATH=/path/to/Qwen3-4B-Base \ DATA_PATH=/path/to/dataset_text_document \ OUTPUT_DIR=/path/to/output \ bash examples/fsdp2/qwen3/pretrain_qwen3_4b_4k_fsdp2_A3.sh The example uses 4 NPUs, a sequence length of 4096, a per-device batch size of 1, and a global batch size of 4 by default. Paths and training arguments can be overridden through environment variables and command-line options. ## How was this patch tested? - Passed Shell syntax validation, YAML parsing, Python compilation, and git diff --check. - Completed 5000 consecutive training steps on 4 Ascend NPUs with a successful process exit. - Loss decreased from 0.04382772 to 0.00364914. - Steps 3-5000 averaged 1.272 seconds/step and 28.830% MFU. - The last 100 steps averaged 1.158 seconds/step, 28.846% MFU, and 14146.63 tokens/s, exceeding the 24% MFU target in the issue. - All 10 distributed checkpoint saves and the final Hugging Face weight export completed successfully. See merge request: Ascend/MindSpeed-LLM!4966 | 24 天前 | |
feat: Add DeepSeek-V4 inference module Co-authored-by: sunjunjie1587<sunjunjie8@huawei.com> # message auto-generated for no-merge-commit merge: !4660 merge master into master feat: Add DeepSeek-V4 inference module Created-by: sunjunjie1587 Commit-by: sunjunjie1587 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds an independent DeepSeek-V4 inference path for MindSpeed-LLM. DeepSeek-V4 uses a model implementation different from the existing GPT inference model, so it cannot cleanly share the original inference.py GPT model provider. This PR keeps the original inference path unchanged and adds a dedicated inference_deepseek4.py entrypoint plus DeepSeek4-specific inference wrappers. Main changes: - Add inference_deepseek4.py as the DeepSeek-V4 inference entrypoint. - Add DeepSeek4-specific inference modules under mindspeed_llm/tasks/inference/: - deepseek4_module.py: inherits the common inference module and isolates DeepSeek4-specific generation behavior. - infer_deepseek4.py: inherits/reuses common inference task logic and isolates DeepSeek4 chat streaming/stdout handling. - Keep common inference modules compatible with the original GPT inference flow, avoiding behavior changes to existing inference.py usage. - Add DeepSeek4 example script: - examples/mcore/deepseek4_flash/generate_deepseek4_flash_4k_A3_ptd.sh - Support DeepSeek4-specific inference requirements, including MTP inference state handling, multi-EOS truncation, streaming delta output, MHC receive-buffer shape adaptation, and DeepSeek4 DSA/G2 attention index alignment. ## Does this PR introduce any user-facing change? Yes. This PR introduces a new DeepSeek-V4 inference entrypoint and example script. Users running DeepSeek-V4 inference should use: bash bash examples/mcore/deepseek4_flash/generate_deepseek4_flash_4k_A3_ptd.sh or directly run: bash torchrun ... inference_deepseek4.py ... Existing GPT/model inference through inference.py remains unchanged. Related files: - inference_deepseek4.py - examples/mcore/deepseek4_flash/generate_deepseek4_flash_4k_A3_ptd.sh - mindspeed_llm/tasks/inference/deepseek4_module.py - mindspeed_llm/tasks/inference/infer_deepseek4.py ## How was this patch tested? This patch was verified with static and syntax checks: bash PYTHONPYCACHEPREFIX=/private/tmp/mindspeed_pycache python3 -m py_compile \ inference_deepseek4.py \ mindspeed_llm/tasks/inference/infer_deepseek4.py \ mindspeed_llm/tasks/inference/deepseek4_module.py \ mindspeed_llm/tasks/inference/infer_base.py \ mindspeed_llm/tasks/inference/module.py bash bash -n examples/mcore/deepseek4_flash/generate_deepseek4_flash_4k_A3_ptd.sh bash git diff --check origin/master..HEAD Expected runtime verification: 1. Prepare DeepSeek-V4 tokenizer and Megatron-format checkpoint. 2. Update TOKENIZER_PATH and CHECKPOINT in examples/mcore/deepseek4_flash/generate_deepseek4_flash_4k_A3_ptd.sh. 3. Launch the script on the target NPU environment. 4. Enter chat prompts and verify that output is streamed normally without repeated full-text printing or delayed large blank output. Constraints and limitations: - DeepSeek-V4 inference should use inference_deepseek4.py, not the generic inference.py. - The provided example is configured for the DeepSeek4 Flash 4K setup and may need TP/PP/EP/CP/path adjustments for other clusters. - Full end-to-end NPU inference needs to be validated in the target hardware environment with the actual checkpoint. 验证记录： INFO:root:=========================================================== INFO:root:1. If you want to quit, please entry one of [q, quit, exit] INFO:root:2. To create new title, please entry one of [clear, new] INFO:root:=========================================================== You >> nihao MindSpeed-LLM: 你好！有什么我可以帮你的吗？😊 You >> nishishei /usr/local/python3.11.15/lib/python3.11/site-packages/torch/distributed/distributed_c10d.py:4631: UserWarning: No device id is provided via init_process_group or barrier . Using the current device set by the user. warnings.warn( # warn only once MindSpeed-LLM: 我是 **DeepSeek，由深度求索公司创造的AI助手！我是 **DeepSeek**，由深度求索公司创造的AI助手！🤖 简单介绍一下自己： -身份纯文本模型知识截止于2025年月能力可以回答问题、写作翻译编程分析文件等特点免费使用支持上传图片PDFWord还能联网搜索（需要手动开启） 版本最新你可以把我当成一个乐于人的智能有什么尽管问我无论是学习工作还是日常闲聊都很乐意帮忙。我是 **DeepSeek**，由深度求索公司创造的AI助手！🤖 简单介绍一下自己： - **身份**：纯文本AI模型，知识截止于2025年5月 - **能力**：可以回答问题、写作、翻译、编程、分析文件等 - **特点**：免费使用，支持上传图片、PDF、Word等文件，还能联网搜索（需要手动开启） - **版本**：DeepSeek最新版模型 你可以把我当成一个乐于助人的智能助手，有什么问题尽管问我！无论是学习、工作还是日常闲聊，我都很乐意帮忙。😊那么今天我可以为你做的吗？ See merge request: Ascend/MindSpeed-LLM!4660 | 3 个月前 | |
[pytorch][feature]Decouple the 'variable-seq-lengths' and 'no-pad-to-seq-lengths' parameters Co-authored-by: mhh001<mahonghao1@huawei.com> # message auto-generated for no-merge-commit merge: !3579 merge master_pad into master [pytorch][feature]Decouple the 'variable-seq-lengths' and 'no-pad-to-seq-lengths' parameters Created-by: mhh111 Commit-by: mhh001 Merged-by: ascend-robot Description: [pytorch][feature]Decouple the 'variable-seq-lengths' and 'no-pad-to-seq-lengths' parameters See merge request: Ascend/MindSpeed-LLM!3579 | 11 个月前 | |
fix: prevent TND CP hang on empty compressed-KV ranks Co-authored-by: ByteDance<lihaozhe9@huawei.com> # message auto-generated for no-merge-commit merge: !5086 merge fix_hang_v5 into master fix: prevent TND CP hang on empty compressed-KV ranks Created-by: lihaozhe777 Commit-by: ByteDance Merged-by: ascend-robot Description: ### What this PR does / why we need it? This PR fixes a backward hang in DeepSeek-V4 CSA training with packed TND sequences and Context Parallelism. A CP rank may contain tokens but have no document segment long enough to produce a complete compressed-KV block. Previously, empty ranks either skipped parts of the indexer/compressor graph or used dummy KV, top-k indices, and weights to enter the fused path. This could cause inconsistent autograd graphs, parameter hooks, gradient buckets, and collective ordering across CP ranks. The fused indexer-loss backward also performs a CP all_gather . When only non-empty ranks entered this collective, training could hang. This patch: - Detects ranks without valid compressed KV before indexer execution. - Falls back to sliding-window attention on empty ranks. - Removes dummy compressed KV, top-k indices, and weights. - Adds zero-gradient anchors for indexer and compressor parameters to preserve gradient hooks and bucket ordering. - Adds a backward communication anchor so empty ranks join the fused indexer-loss CP collective. - Records zero indexer loss on empty ranks to keep loss tracker keys aligned. - Extracts the shared TND token-count collective into gather_tnd_local_num() . - Updates only the required tests/mindspeed_llm CI mirror files. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? - Ran Python syntax checks with py_compile . - Ran git diff --check . - Ran repository pre-commit checks, including Ruff, Pylint, Bandit, typos, and secret scanning. - Verified that only the required DeepSeek-V4 CI mirror files were changed. - Reviewed both empty-rank and non-empty-rank control flow for TND packed sequences. See merge request: Ascend/MindSpeed-LLM!5086 | 24 天前 | |
fix: prevent TND CP hang on empty compressed-KV ranks Co-authored-by: ByteDance<lihaozhe9@huawei.com> # message auto-generated for no-merge-commit merge: !5086 merge fix_hang_v5 into master fix: prevent TND CP hang on empty compressed-KV ranks Created-by: lihaozhe777 Commit-by: ByteDance Merged-by: ascend-robot Description: ### What this PR does / why we need it? This PR fixes a backward hang in DeepSeek-V4 CSA training with packed TND sequences and Context Parallelism. A CP rank may contain tokens but have no document segment long enough to produce a complete compressed-KV block. Previously, empty ranks either skipped parts of the indexer/compressor graph or used dummy KV, top-k indices, and weights to enter the fused path. This could cause inconsistent autograd graphs, parameter hooks, gradient buckets, and collective ordering across CP ranks. The fused indexer-loss backward also performs a CP all_gather . When only non-empty ranks entered this collective, training could hang. This patch: - Detects ranks without valid compressed KV before indexer execution. - Falls back to sliding-window attention on empty ranks. - Removes dummy compressed KV, top-k indices, and weights. - Adds zero-gradient anchors for indexer and compressor parameters to preserve gradient hooks and bucket ordering. - Adds a backward communication anchor so empty ranks join the fused indexer-loss CP collective. - Records zero indexer loss on empty ranks to keep loss tracker keys aligned. - Extracts the shared TND token-count collective into gather_tnd_local_num() . - Updates only the required tests/mindspeed_llm CI mirror files. ### Does this PR introduce any user-facing change? No. ### How was this patch tested? - Ran Python syntax checks with py_compile . - Ran git diff --check . - Ran repository pre-commit checks, including Ruff, Pylint, Bandit, typos, and secret scanning. - Verified that only the required DeepSeek-V4 CI mirror files were changed. - Reviewed both empty-rank and non-empty-rank control flow for TND packed sequences. See merge request: Ascend/MindSpeed-LLM!5086 | 24 天前 | |
fix(pytorch)：fix kvallgather for pipeline. Co-authored-by: yanzhixiao<yanzhixiao@h-partners.com> # message auto-generated for no-merge-commit merge: !4409 merge bugfix-pipeline-0414 into master fix(pytorch)：fix kvallgather for pipeline. Created-by: yanzhixiao23 Commit-by: yanzhixiao Merged-by: ascend-robot Description: ## What this PR does / why we need it? Fix kvallgather for pipeline. ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed-LLM!4409 | 5 个月前 | |
feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Co-authored-by: wj<wangjin230@huawei.com> # message auto-generated for no-merge-commit merge: !4878 merge fsdp2-msprobe into master feat(debug): support msProbe and model I/O tracing for FSDP2 and Megatron Created-by: gcw_RxnYoBVv Commit-by: wj Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR adds optional precision-data collection support for both FSDP2 and Megatron training backends. ## Does this PR introduce any user-facing change? Yes. The feature is disabled by default and does not affect existing training jobs unless explicitly enabled. For FSDP2, users can configure the features in the training YAML: yaml training: msprobe: true msprobe_config_path: /absolute/path/to/msprobe_config.json model_io_trace: true model_io_trace_config_path: /absolute/path/to/model_io_trace_config.json For Megatron, the corresponding arguments are: bash --msprobe \ --msprobe-config-path /absolute/path/to/msprobe_config.json \ --model-io-trace \ --model-io-trace-config-path /absolute/path/to/model_io_trace_config.json Related documentation: - docs/zh/pytorch/tools/msprobe.md - docs/zh/pytorch/tools/model_io_trace.md - docs/zh/pytorch/features/fsdp2/arguments.md ## How was this patch tested? After enabling the function, can the relevant files be generated after the training script is completed See merge request: Ascend/MindSpeed-LLM!4878 | 1 个月前 | |
feat(pytorch): integrate DeepSeek V4 compressed-attention context parallel Co-authored-by: Liz<lizhi166@huawei.com> # message auto-generated for no-merge-commit merge: !5017 merge stream-master-0902 into master feat(pytorch): integrate DeepSeek V4 compressed-attention context parallel Created-by: Liz_ Commit-by: Liz Merged-by: ascend-robot Description: ## What this PR does / why we need it? DS V4 CP adaptor ## Does this PR introduce any user-facing change? NAN ## How was this patch tested? https://wiki.huawei.com/domains/64729/wiki/343185/WIKI2026090512720782 See merge request: Ascend/MindSpeed-LLM!5017 | 1 个月前 | |
Qwen3-235B model pre-training upgrade 018 adaptation Co-authored-by: cjy840282<chenjingyi9@huawei.com> # message auto-generated for no-merge-commit merge: !5001 merge upgrade into master Qwen3-235B model pre-training upgrade 018 adaptation Created-by: cjy840282 Commit-by: cjy840282 Merged-by: ascend-robot Description: ## What this PR does / why we need it? This PR implements adaptation for the Qwen3-235B model to the pre-training upgrade 018. ## Does this PR introduce any user-facing change? Users will not perceive any adaptation modifications. ## How was this patch tested? Accuracy, performance, memory usage, and long-running stability targets are met. See merge request: Ascend/MindSpeed-LLM!5001 | 1 个月前 |