已合并
fixing experimental option and ReduceScatter BIRS #2185
Davydov_Danil创建于 7月23日
fixing experimental option and ReduceScatter BIRS #2185
已合并
Davydov_Danil创建于 7月23日
Davydov_Danil
7月23日

Description

Fixes:

  • build.sh 的 build_device() 未转发 ENABLE_EXPERIMENTAL,导致 experimental BIRS 执行器未编入 device 库,运行时 ReduceScatter 失败
  • ReduceScatter BIRS 多层忽略通信返回值导致失败后仍继续执行

Build-system changes required for the first fix:

  • Third-party dependency pin bump: cann-cmake master-044 -> master-051 in cmake/fetch_cann_cmake.cmake
    (URL_HASH 2d8827b3... -> f50ea0bf...). Reason: only master-051's add_cann_device_project supports
    extra-argument passthrough (cmake_parse_arguments(CANN "" "" "EXT_ARGS" ${ARGN}) +
    list(APPEND EP_CMAKE_ARGS ${CANN_EXT_ARGS}) in function/prepare.cmake); master-044 has no EXT_ARGS.
    This is the prerequisite for CMakeLists.txt: add_cann_device_project(hccl EXT_ARGS -D ENABLE_EXPERIMENTAL=${ENABLE_EXPERIMENTAL}),
    i.e. the flag reaches the device ExternalProject without keeping an in-repo override copy of add_cann_device_project.
    Pin verified: SHA256 of https://raw.gitcode.com/cann/cmake/archive/refs/heads/master-051.tar.gz
    == f50ea0bf7f80ce9e664f26693252aee8d2a42438a80bc2cd335741dc4b3b834f.
    044 -> 051 side effects reviewed (new __cann_check_build_env() gate: gcc >= 7.3.0 + python3;
    convert_dependencies_to_package_formats renamed to a private __cann_ helper; device EP no longer forwards
    -D VERSION_INFO): hccl does not call the renamed function, and cmake/device/CMakeLists.txt includes
    version.cmake directly, so host and device builds are unaffected.
  • cmake/device/CMakeLists.txt: added option(ENABLE_EXPERIMENTAL "Enable experimental features" OFF) so the
    standalone device project accepts the forwarded flag and gates add_subdirectory(experimental/ops).
  • Removed the now-redundant if(NOT ENABLE_EXPERIMENTAL) return() guards from 6 leaf CMakeLists under
    experimental/ops (op_common, op_common/template, op_common/topo, reduce_scatter,
    reduce_scatter/birs, reduce_scatter/birs/template), per review. Gating stays single-sourced:
    the guard in experimental/ops/CMakeLists.txt plus the two add_subdirectory call sites
    (root CMakeLists.txt for host, cmake/device/CMakeLists.txt for device); leaf target_sources(hccl ...)
    remains protected by if(TARGET hccl), so the device build (no hccl target) is unaffected.
  • experimental/ops/all_reduce/example: GetAlgNetMeta() signature adapted to the current base-class
    interface (added const OpParam& param).

https://gitcode.com/cann/hccl/issues/324
https://gitcode.com/cann/hccl/issues/380

Testing

Testing with the hccl-checker tool via ST tests "/test/st/algorithm/testcase/reduce_scatter_testcase_a3.cc"
Single-node: ReduceScatterBirs was deployed on one Atlas A3 node with RankSize set between 4 and 16.
Example of ReduceScatterBIRS output from hccl_test for RankSize 16:
image.png

Documentation Update

  • docs/zh/build/build.md and docs/en/build/build.md: third-party software table, cann-cmake row
    master-044 -> master-051. Download source switched from the cann-3rd OBS bucket
    (cmake-master-051.tar.gz is not published there — HTTP 403) to the raw.gitcode.com archive, whose
    SHA256 matches the pin declared in cmake/fetch_cann_cmake.cmake.

Type Label

likedislike
Pull Request已成功合入, 合并人@CANN-robot
(感谢 Davydov_Danil 的贡献)
DDavydov_Danil
7月23日 关联了issue:[Bug-Report|缺陷反馈]: 【2026 HCCL通信库创新大赛-粤港澳赛区】【XDU】ReduceScatter BIRS 多层忽略通信返回值导致失败后仍继续执行,[Bug-Report|缺陷反馈]: build.sh 的 build_device() 未转发 ENABLE_EXPERIMENTAL,导致 experimental BIRS 执行器未编入 device 库,运行时 ReduceScatter 失败
atomgit-bot
atomgit-bot
7月23日 评论:

变更摘要

此 PR 修复了两个问题:一是 build.sh 中 build_device() 函数未将 ENABLE_EXPERIMENTAL 环境变量转发给 CMake,导致 experimental BIRS 执行器未被编译进 device 库,运行时 ReduceScatter 失败;二是 ReduceScatterBIRS 及其跨服务器版本 ReduceScatterBIRSInter 中多处关键函数调用忽略了通信返回值(未用 CHK_RET 宏检查),导致通信失败后仍继续执行。修复方式是在 build.sh 中补充 -DENABLE_EXPERIMENTAL 参数传递,并在两个算法的 RunAsync、HCCSProcessMainLoop/HCCSIntraStep、SIOProcessMainLoop/SIOIntraStep、InterServer 等核心流程中对 HcommWriteOnThread、HcommWriteReduceOnThread、Preprocess、LocalReduceCCLToCCL、FinalStep 等调用统一添加 CHK_RET 返回值检查。

主要改动

  • build.sh 中补充 experimental 编译选项转发:在 build_device() 函数中新增对 ENABLE_EXPERIMENTAL 环境变量的读取,将其转换为 DEVICE_ENABLE_EXPERIMENTAL 并通过 -DENABLE_EXPERIMENTAL 传递给 CMake,确保 experimental 特性被正确编译进 device 库。
  • ReduceScatterBIRS::RunAsync 中补充返回值检查:对 Preprocess、HCCSProcessMainLoop、SIOProcessMainLoop、LocalCopyMainLoop、FinalStep 的调用添加 CHK_RET 宏包裹,防止通信步骤失败后静默继续执行。
  • ReduceScatterBIRS 底层通信调用补充返回值检查:在 HCCSProcessMainLoop 中对 HcommWriteOnThread、在 SIOProcessMainLoop 中对 HcommWriteReduceOnThread 添加 CHK_RET,确保 HCCS 写入和 SIO 归约写入的错误能被捕获。
  • ReduceScatterBIRSInter 中同步补充返回值检查:在 RunAsync、HCCSIntraStep、SIOIntraStep、InterServer 中对 Preprocess、LocalCopyPreproc、HcommWriteOnThread、HcommWriteReduceOnThread、LocalReduceCCLToCCL、PreprocInterServer、InterServer 等调用统一添加 CHK_RET,使跨服务器版本与单服务器版本行为一致。
likedislike
不准确?
atomgit-bot
atomgit-bot
7月23日 评论:

代码审查

发现 2 个问题,均为中低优先级建议。

类型 数量
🔴 阻塞 0
🟡 建议 2

💬 仅评论

likedislike
不准确?
CANN-robotCANN-robot成员
7月23日 添加了label:cann-cla/yes
CANN-robot
CANN-robot成员
7月23日 评论:

CLA Signature Pass

Davydov_Danil, thanks for your pull request. All authors of the commits have signed the CLA. 👍

likedislike
此处折叠了401条消息 查看更多
CANN-robotCANN-robot成员
23 天前 添加了label:ci-pipeline-running
CANN-robotCANN-robot成员
23 天前 删除了label:ci-pipeline-running
CANN-robotCANN-robot成员
23 天前 添加了label:ci-pipeline-passed
CANN-robotCANN-robot成员
23 天前 关闭了关联的issue
CANN-robotCANN-robot成员
23 天前 合入了pull request