sort_kernel

ascend-robotfeat: support W4A8-QAT

035ab317创建于 7月15日历史提交

文件	最后提交记录	最后更新时间
context_parallel	fix(torch/cp): use sbnd format before all2all Co-authored-by: clc2025<chenlucong@huawei.com> # message auto-generated for no-merge-commit merge: !3282 merge fixbug_ulysses_tnd into master fix(torch/cp): use sbnd format before all2all Created-by: clc2025 Commit-by: clc2025 Merged-by: ascend-robot Description: fixbug for ulysses tnd See merge request: Ascend/MindSpeed!3282	6 个月前
data_parallel	!2716 fix: async_log_allreduce Merge pull request !2716 from 邓佳/core_r0.12.1_fix_v3	1 年前
datasets	Add offline pad_data Co-authored-by: wuweiqiang24<wuweiqiang11@huawei.com> # message auto-generated for no-merge-commit merge: !2938 merge revise_preprocess_data into master Add offline pad_data Created-by: wuweiqiang24 Commit-by: wuweiqiang24 Merged-by: ascend-robot Description: 增加离线预处理pack数据集功能，可提前将数据padding到2\*CP倍，在线使用CP功能时可节约padding部分耗时 * 精度与非离线padding版本存在一定差异  * Llama2-7b，单机16k，GBS=8场景下，性能提升4.8%  See merge request: Ascend/MindSpeed!2938	9 个月前
dist_checkpointing	perf(verl ckpt): ckpt load and save acceleration Co-authored-by: 李鸣沼<lmztju@126.com> # message auto-generated for no-merge-commit merge: !3074 merge verl_load_and_save_ckpt into master perf(verl ckpt): ckpt load and save acceleration Created-by: lmztju Commit-by: 李鸣沼;l30057177 Merged-by: ascend-robot Description: **测试场景1：** verl+megatron后端dapo-qwen3-30b 910A3双机加载和保存ckpt 本地存储  See merge request: Ascend/MindSpeed!3074	8 个月前
distributed	docs:fix docs/zh mistakes Co-authored-by: Keilo_W<wangkaiyu11@h-partners.com> # message auto-generated for no-merge-commit merge: !3318 merge master into master docs:fix docs/zh mistakes Created-by: Keilo_W Commit-by: Keilo_W Merged-by: ascend-robot Description: 修改了一些被误操作的注释及代码 See merge request: Ascend/MindSpeed!3318	5 个月前
fusions	feat: support W4A8-QAT Co-authored-by: xusiyang<xusiyang2@huawei.com> # message auto-generated for no-merge-commit merge: !3664 merge master into master feat: support W4A8-QAT Created-by: weixin_44492126 Commit-by: xusiyang Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本PR 新增了对 W4A8-QAT（权重MXFP4/激活 MXFP8 量化感知训练，仅限 MoE 场景），并量化粒度分别支持：32(标准MXFP4量化粒度)/128(deepseek-V4采用量化粒度） ## Does this PR introduce any user-facing change? Yes. 用户可以通过在训练命令中添加以下参数来启用W4A8功能，默认量化粒度32： --qat-scheme w4a8-moe-only --qat-quant-block-size 128 ## How was this patch tested? 添加 --qat-scheme w4a8-moe-only 参数启动量化粒度为32的W4A8-QAT训练，添加--qat-quant-block-size 128启动量化粒度为128的W4A8-QAT训练 实现方案和实验结果如下：https://wiki.huawei.com/domains/171785/wiki/358154/WIKI2026071511870516 昇腾 950DT上测试通过：基于减层DeepSeek-V4，分别训练100step对比BF16 loss误差：量化粒度为32时loss误差在0.4%，量化粒度为128时loss误差在0.79% See merge request: Ascend/MindSpeed!3664	1 个月前
hccl_buffer	fix hccl buffer errors for verl cases Co-authored-by: quancs001<quancs@qq.com> # message auto-generated for no-merge-commit merge: !3478 merge fix_hccl_buffer_for_verl into master fix hccl buffer errors for verl cases Created-by: quancs001 Commit-by: quancs001 Merged-by: ascend-robot Description: What this PR does / why we need it? When running RL exps with verl, several errors are raised, e.g.: 1. megatron.training.get_args raises an exception 2. the args for hccl_buffer with ";" could not be parsed by hydra, and the error LexerNoViableAltException is raised. This PR is proposed to solve the errors. Does this PR introduce any user-facing change? No. How was this patch tested? The code is tested and verified locally. See merge request: Ascend/MindSpeed!3478	3 个月前
megatron_basic	fix: TE LayerNormLinear init weight order align NVTE Co-authored-by: clc2025<chenlucong@huawei.com> # message auto-generated for no-merge-commit merge: !3569 merge fix_lnliner_initweight into master fix: TE LayerNormLinear init weight order align NVTE Created-by: clc2025 Commit-by: clc2025 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 具体查看关联issue ## Does this PR introduce any user-facing change? NA ## How was this patch tested? 基于脚本用例，从GPU上保存权重，NPU加载权重后断点续训，精度能对齐且无功能报错 See merge request: Ascend/MindSpeed!3569	2 个月前
memory	Add swap layer input Co-authored-by: JialiZheng<jializheng@huawei.com> # message auto-generated for no-merge-commit merge: !3529 merge master into master Add swap layer input Created-by: JialiZheng1 Commit-by: JialiZheng Merged-by: ascend-robot Description: Add swap input RFC：https://gitcode.com/Ascend/MindSpeed/issues/176 验证wiki：https://wiki.huawei.com/wiki/WIKI2026060411356524 See merge request: Ascend/MindSpeed!3529	2 个月前
models	quant fp8 optimizer	9 个月前
multi_modal	disttrain intervl2 适配开箱 Co-authored-by: gcw_amOUPDs9<fuyuefeng@huawei.com> # message auto-generated for no-merge-commit merge: !2915 merge master into master disttrain intervl2 适配开箱 Created-by: gcw_amOUPDs9 Commit-by: gcw_amOUPDs9 Merged-by: ascend-robot Description: disttrain intervl2 适配开箱 disttrain intervl2 适配开箱 See merge request: Ascend/MindSpeed!2915	10 个月前
optimizer	fix: TE LayerNormLinear init weight order align NVTE Co-authored-by: clc2025<chenlucong@huawei.com> # message auto-generated for no-merge-commit merge: !3569 merge fix_lnliner_initweight into master fix: TE LayerNormLinear init weight order align NVTE Created-by: clc2025 Commit-by: clc2025 Merged-by: ascend-robot Description: ## What this PR does / why we need it? 具体查看关联issue ## Does this PR introduce any user-facing change? NA ## How was this patch tested? 基于脚本用例，从GPU上保存权重，NPU加载权重后断点续训，精度能对齐且无功能报错 See merge request: Ascend/MindSpeed!3569	2 个月前
performance	!2635 安全：文件路径校验/权限 Merge pull request !2635 from glhyy/secmaster	1 年前
pipeline_parallel	feat: add custom pp layout Co-authored-by: wuweiqiang24<wuweiqiang11@huawei.com> # message auto-generated for no-merge-commit merge: !3496 merge add_pp_layout into master feat: add custom pp layout Created-by: wuweiqiang24 Commit-by: wuweiqiang24 Merged-by: ascend-robot Description: 新增pipeline-model-parallel-layout功能，支持自定义PP每个stage的层排布 验证链接：https://wiki.huawei.com/domains/137239/wiki/268925/WIKI2026052611233549 issue: https://gitcode.com/Ascend/MindSpeed/issues/166 See merge request: Ascend/MindSpeed!3496	3 个月前
qat	feat(qat): add W4A4 MXFP4 quantization-aware training support Co-authored-by: n_nddddd<lanshangwei1@huawei.com> # message auto-generated for no-merge-commit merge: !3614 merge feat_qat_w4a4_1 into master feat(qat): add W4A4 MXFP4 quantization-aware training support Created-by: n_nddddd Commit-by: n_nddddd Merged-by: ascend-robot Description: title: "feat: add w4a4 qat feature" labels: ["feat"] assignees: lanshangwei What this PR does / why we need it? add feature of qat w4a4 Does this PR introduce any user-facing change? no, just add a new qat type How was this patch tested? 实验报告 https://wiki.huawei.com/domains/159368/wiki/325164/WIKI2026070211708534 UT运行报告： ============================= test session starts ============================== platform linux -- Python 3.10.19, pytest-9.0.2, pluggy-1.6.0 -- /home/anaconda3/envs/llm_lsw/bin/python cachedir: .pytest_cache rootdir: /home/l00611484/workspace/MindSpeed plugins: mock-3.15.1, jaxtyping-0.3.4, hydra-core-1.3.2, anyio-4.12.0 collecting ... collected 17 items tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_output_shape PASSED [ 5%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_output_not_nan PASSED [ 11%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_deterministic PASSED [ 17%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_dtype_preserved PASSED [ 23%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_zero_tensor PASSED [ 29%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_small_values PASSED [ 35%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_ste_backward_exists PASSED [ 41%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4FakeQuantization::test_ste_grad_values PASSED [ 47%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearForward::test_forward_shape PASSED [ 52%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearForward::test_forward_with_bias PASSED [ 58%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearForward::test_forward_not_nan PASSED [ 64%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearForward::test_forward_differentiable PASSED [ 70%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearBackward::test_backward_shapes PASSED [ 76%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearBackward::test_backward_not_nan PASSED [ 82%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestW4A4LinearBackward::test_dw_uses_quantized_input PASSED [ 88%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestIntegration::test_end_to_end PASSED [ 94%] tests_extend/unit_tests/features/qat/test_w4a4_core_functions.py::TestIntegration::test_gradient_flow PASSED [100%] See merge request: Ascend/MindSpeed!3614	2 个月前
qos	feat: add A2G3 device type support across codebase Co-authored-by: fanlu5<fanlu5@huawei.com> # message auto-generated for no-merge-commit merge: !3587 merge master into master feat: add A2G3 device type support across codebase Created-by: fanlu5 Commit-by: fanlu5 Merged-by: ascend-robot Description: ## What this PR does / why we need it? feat: add A2G3 device type support across codebase. 添加A2G3设备类型的适配支持，将其与现有的910B设备同等对待，覆盖版本检测、搜索空间裁剪、通信硬件信息、MoE相关逻辑等多个模块 ## Does this PR introduce any user-facing change? NA ## How was this patch tested? NA See merge request: Ascend/MindSpeed!3587	2 个月前
tensor_parallel	feature(fp8): te checkpoint Co-authored-by: Muu<koimuu@163.com> # message auto-generated for no-merge-commit merge: !3162 merge feature_checkpoint into master feature(fp8): te checkpoint Created-by: Muuyo Commit-by: Muu Merged-by: ascend-robot Description: 1. 引入 te checkpoint消除重计算中冗余的量化操作 2. refactor(blockwise): 删除128*128的blockwise策略, 保留1 * 128|128 * 128策略替换 3. perf(hif8): 删除多余的cast 4. fix(delayed): 修复delayed算法 5. refactor(recipe 2x): 重构blockwise和mxfp8策略数据存取, 简化后续算子适配 6. 消除字符串字面量, 采用枚举替代 验证报告: https://wiki.huawei.com/domains/76578/wiki/233229/WIKI202601139775970 See merge request: Ascend/MindSpeed!3162	7 个月前
transformer	feat: support W4A8-QAT Co-authored-by: xusiyang<xusiyang2@huawei.com> # message auto-generated for no-merge-commit merge: !3664 merge master into master feat: support W4A8-QAT Created-by: weixin_44492126 Commit-by: xusiyang Merged-by: ascend-robot Description: ## What this PR does / why we need it? 本PR 新增了对 W4A8-QAT（权重MXFP4/激活 MXFP8 量化感知训练，仅限 MoE 场景），并量化粒度分别支持：32(标准MXFP4量化粒度)/128(deepseek-V4采用量化粒度） ## Does this PR introduce any user-facing change? Yes. 用户可以通过在训练命令中添加以下参数来启用W4A8功能，默认量化粒度32： --qat-scheme w4a8-moe-only --qat-quant-block-size 128 ## How was this patch tested? 添加 --qat-scheme w4a8-moe-only 参数启动量化粒度为32的W4A8-QAT训练，添加--qat-quant-block-size 128启动量化粒度为128的W4A8-QAT训练 实现方案和实验结果如下：https://wiki.huawei.com/domains/171785/wiki/358154/WIKI2026071511870516 昇腾 950DT上测试通过：基于减层DeepSeek-V4，分别训练100step对比BF16 loss误差：量化粒度为32时loss误差在0.4%，量化粒度为128时loss误差在0.79% See merge request: Ascend/MindSpeed!3664	1 个月前
ttp	feat: add last word feature Co-authored-by: Zhaiwenxuan<zhaiwenxuan4@h-partners.com> # message auto-generated for no-merge-commit merge: !3542 merge master into master feat: add last word feature Created-by: Zhaiwenxuan Commit-by: Zhaiwenxuan Merged-by: ascend-robot Description: ## What this PR does / why we need it? 新增适配verl训练场景的临终遗言功能 ## Does this PR introduce any user-facing change? YES ++actor_rollout_ref.actor.megatron.override_transformer_config.ttp_enabled=True 用户可通过增加++actor_rollout_ref.actor.megatron.override_transformer_config.ttp_enabled=True，开启适配verl训练场景的临终遗言功能 ## How was this patch tested? https://wiki.huawei.com/domains/181442/wiki/383165/WIKI2026060911405364 See merge request: Ascend/MindSpeed!3542	2 个月前
__init__.py	!359 change ascendspeed to mindspeed Merge pull request !359 from 邓佳/master	2 年前
fp8_utils.py	feat: mxfp8-32x32 quant Co-authored-by: kyle_zhangchi<zhangchi158@huawei.com> # message auto-generated for no-merge-commit merge: !3471 merge feat_mxfp8-32x32 into master feat: mxfp8-32x32 quant Created-by: kyle_zhangchi Commit-by: kyle_zhangchi Merged-by: ascend-robot Description: ## What this PR does / why we need it? 在Megatron框架下新增mxfp8-32x32量化算子，降低权重显存占用 ## Does this PR introduce *any* user-facing change? --fp8-recipe新增mxfp8-32x32选项 https://gitcode.com/Ascend/MindSpeed/commit/e065cbca6873bfc02661d088b07d90224333e87d?ref=feat_mxfp8-32x32&prId=3471 ## How was this patch tested? 验证文档 https://wiki.huawei.com/domains/170864/wiki/367830/WIKI2026051111046509 See merge request: Ascend/MindSpeed!3471	3 个月前
mindspeed_parallel_group.py	!699 2d张量并行 Merge pull request !699 from liujianxing/2d_tensor_0824	1 年前
parallel_state.py	!2552 [fix bug] hccl_buff cannt be zero Merge pull request !2552 from 李宝奎/master_wtd	1 年前
simple_parallel_cfg.py	!699 2d张量并行 Merge pull request !699 from liujianxing/2d_tensor_0824	1 年前
singleton_meta.py	!699 2d张量并行 Merge pull request !699 from liujianxing/2d_tensor_0824	1 年前
tensor_parallel_y_union_cp.py	!699 2d张量并行 Merge pull request !699 from liujianxing/2d_tensor_0824	1 年前
training.py	fixbug for auto_settings Co-authored-by: wuweiqiang24<wuweiqiang11@huawei.com> # message auto-generated for no-merge-commit merge: !3003 merge auto_settings into master fixbug for auto_settings Created-by: wuweiqiang24 Commit-by: wuweiqiang24 Merged-by: ascend-robot Description: 1. fixbug for auto_settings 2. add readme See merge request: Ascend/MindSpeed!3003	9 个月前
weight_grad_store.py	!1216 安全排查修改 Merge pull request !1216 from huangzhenyu/master-icsl-no-ootb	1 年前