Merged
feat: add distributed ops AllGatherMatmul and MatmulReduceScatter (MC2) #656
hedongdongAdd on May 18
feat: add distributed ops AllGatherMatmul and MatmulReduceScatter (MC2) #656
Merged
Pull Request has been successfully merged by MindSpore-Bot.
(Thank you for hedongdong's contribution.)May 18 Created pull request, commit 54554447
May 18 Added label: mindspore-cla/yes
司小南(机器人)
May 18 Comments:
May 18 Comments:
@david-he91, 当前/check-pr未通过,原因如下:
以下Pull Request描述检查项未通过:
存在不符合模板的选项: 设计:AllGatherMatmul / MatmulReduceScatter 分布式 layout 推导逻辑已覆盖所有合法分片组合
存在不符合模板的选项: 测试:UT 25 cases + ST 9 cases,覆盖 normalize / preprocess / infer_layout 及分布式场景
存在不符合模板的选项: 接口:通过 YAML 注册,对上层调用方无破坏性变更
存在不符合模板的选项: 约束文档:trans_x2、gather_output 语义、k 维约束已在 docstring 中注明
部分检查项缺失 请重新使用模板
模板中'Test Plan and Test Result' 信息为空,请补充对应信息。
以下issue检查项未通过:
Pull Request未关联issue
请修改好上述检查错误后,重新使用/check-pr触发检查。


May 18 Added label: pr-check-fail
May 18 Added label: no-pass-all-review
76 messages are collapsed here。 See more
May 19 Approval passed
MindSpore-Bot
May 19 Comments:
May 19 Comments:
May 19 Approval passed
May 19 Removed label: no-pass-all-review
May 19 Merged pull request, merge node SHA: 51c236bd893d04e7d4325c38d1682b062ab5b78d
What type of PR is this?
/kind feature
What does this PR do / why do we need it:
为 HyperParallel 接入 Matrix Computation & Communication通算融合算子的分布式调度逻辑,包含:
mindspore.ops.all_gather_matmul(AllGatherMatmul)mindspore.ops.matmul_reduce_scatter(MatmulReduceScatter)CANN MC2 算子在内核内部自行完成 AllGather / ReduceScatter 通信,不需要 HyperParallel 插入额外通信原语。HyperParallel 的作用是:
preprocess()的to_local()提取本地 tensor 后再传给 CANN。infer_layout()基于输入分片方式推断输出 tensor 的分布状态,使下游算子能正确组合分布式计算图。AllGatherMatmul
输出 layout 推断:
Partial(sum)(与LinearDistributedOpcontract_dim 分片语义一致)MatmulReduceScatter
输出 layout 推断:
测试覆盖
test_parallel_all_gather_matmul.py(15 cases)test_parallel_matmul_reduce_scatter.py(12 cases)all_gather_matmul_shard_in_python.py(7 cases)matmul_reduce_scatter_shard_in_python.py(5 cases)Which issue(s) this PR fixes:
Fixes #152
Test Plan and Test result:What scenarios were tested, and what were the verification results(Function, performance, reliability, etc.):
UT(单元测试)
tests/ut/core/shard/ops/test_parallel_all_gather_matmul.py— 15 cases,覆盖参数归一化、YAML 注册、infer_layout(replicate/shard/trans_x2/2D-mesh/k 分片带 Partial)、错误处理(Partial 输入、k-placement 不匹配、m 维多 mesh 联合分片)tests/ut/core/shard/ops/test_parallel_matmul_reduce_scatter.py— 12 cases,覆盖参数归一化、YAML 注册、infer_layout(TP/DP+TP/n-shard/trans_x2/联合 tuple m 分片)、错误处理(Partial 输入、k Replicate、k layout 不一致)ST(系统测试 - distributed, MindSpore)
tests/mindspore/st/shard/ops/test_all_gather_matmul_shard_in_python.py— 3 组:group1(4×2 卡)、group2(2×4 卡,含 k 分片场景)、group3(1×8 卡,mnk 三轴全切)tests/mindspore/st/shard/ops/test_matmul_reduce_scatter_shard_in_python.py— 3 组:group1(3×2 卡)、group2(1×4 卡)、group3(1×8 卡,mnk 三轴全切)验证结果
Self-checklist:(请自检,在[ ]内打上x,我们将检视你的完成情况,否则会导致pr无法合入)