已合并
fixing experimental option and ReduceScatter BIRS #2185
Davydov_Danil创建于 7月23日
fixing experimental option and ReduceScatter BIRS #2185
已合并
Pull Request已成功合入, 合并人@CANN-robot
(感谢 Davydov_Danil 的贡献)DDavydov_Danil
7月23日 关联了issue:[Bug-Report|缺陷反馈]: 【2026 HCCL通信库创新大赛-粤港澳赛区】【XDU】ReduceScatter BIRS 多层忽略通信返回值导致失败后仍继续执行,[Bug-Report|缺陷反馈]: build.sh 的 build_device() 未转发 ENABLE_EXPERIMENTAL,导致 experimental BIRS 执行器未编入 device 库,运行时 ReduceScatter 失败
7月23日 关联了issue:[Bug-Report|缺陷反馈]: 【2026 HCCL通信库创新大赛-粤港澳赛区】【XDU】ReduceScatter BIRS 多层忽略通信返回值导致失败后仍继续执行,[Bug-Report|缺陷反馈]: build.sh 的 build_device() 未转发 ENABLE_EXPERIMENTAL,导致 experimental BIRS 执行器未编入 device 库,运行时 ReduceScatter 失败
atomgit-bot
7月23日 评论:
7月23日 评论:
变更摘要
此 PR 修复了两个问题:一是 build.sh 中 build_device() 函数未将 ENABLE_EXPERIMENTAL 环境变量转发给 CMake,导致 experimental BIRS 执行器未被编译进 device 库,运行时 ReduceScatter 失败;二是 ReduceScatterBIRS 及其跨服务器版本 ReduceScatterBIRSInter 中多处关键函数调用忽略了通信返回值(未用 CHK_RET 宏检查),导致通信失败后仍继续执行。修复方式是在 build.sh 中补充 -DENABLE_EXPERIMENTAL 参数传递,并在两个算法的 RunAsync、HCCSProcessMainLoop/HCCSIntraStep、SIOProcessMainLoop/SIOIntraStep、InterServer 等核心流程中对 HcommWriteOnThread、HcommWriteReduceOnThread、Preprocess、LocalReduceCCLToCCL、FinalStep 等调用统一添加 CHK_RET 返回值检查。
主要改动
build.sh中补充 experimental 编译选项转发:在build_device()函数中新增对ENABLE_EXPERIMENTAL环境变量的读取,将其转换为DEVICE_ENABLE_EXPERIMENTAL并通过-DENABLE_EXPERIMENTAL传递给 CMake,确保 experimental 特性被正确编译进 device 库。ReduceScatterBIRS::RunAsync中补充返回值检查:对Preprocess、HCCSProcessMainLoop、SIOProcessMainLoop、LocalCopyMainLoop、FinalStep的调用添加CHK_RET宏包裹,防止通信步骤失败后静默继续执行。ReduceScatterBIRS底层通信调用补充返回值检查:在HCCSProcessMainLoop中对HcommWriteOnThread、在SIOProcessMainLoop中对HcommWriteReduceOnThread添加CHK_RET,确保 HCCS 写入和 SIO 归约写入的错误能被捕获。ReduceScatterBIRSInter中同步补充返回值检查:在RunAsync、HCCSIntraStep、SIOIntraStep、InterServer中对Preprocess、LocalCopyPreproc、HcommWriteOnThread、HcommWriteReduceOnThread、LocalReduceCCLToCCL、PreprocInterServer、InterServer等调用统一添加CHK_RET,使跨服务器版本与单服务器版本行为一致。


不准确?
atomgit-bot
7月23日 评论:
7月23日 评论:
7月23日 添加了label:cann-cla/yes
CANN-robot
7月23日 评论:
7月23日 评论:
此处折叠了401条消息 查看更多
23 天前 添加了label:ci-pipeline-running
23 天前 删除了label:ci-pipeline-running
23 天前 添加了label:ci-pipeline-passed
23 天前 关闭了关联的issue
23 天前 合入了pull request
Description
Fixes:
Build-system changes required for the first fix:
cmake/fetch_cann_cmake.cmake(URL_HASH
2d8827b3...->f50ea0bf...). Reason: only master-051'sadd_cann_device_projectsupportsextra-argument passthrough (
cmake_parse_arguments(CANN "" "" "EXT_ARGS" ${ARGN})+list(APPEND EP_CMAKE_ARGS ${CANN_EXT_ARGS})infunction/prepare.cmake); master-044 has noEXT_ARGS.This is the prerequisite for
CMakeLists.txt:add_cann_device_project(hccl EXT_ARGS -D ENABLE_EXPERIMENTAL=${ENABLE_EXPERIMENTAL}),i.e. the flag reaches the device ExternalProject without keeping an in-repo override copy of
add_cann_device_project.Pin verified: SHA256 of https://raw.gitcode.com/cann/cmake/archive/refs/heads/master-051.tar.gz
==
f50ea0bf7f80ce9e664f26693252aee8d2a42438a80bc2cd335741dc4b3b834f.044 -> 051 side effects reviewed (new
__cann_check_build_env()gate: gcc >= 7.3.0 + python3;convert_dependencies_to_package_formatsrenamed to a private__cann_helper; device EP no longer forwards-D VERSION_INFO): hccl does not call the renamed function, andcmake/device/CMakeLists.txtincludesversion.cmakedirectly, so host and device builds are unaffected.cmake/device/CMakeLists.txt: addedoption(ENABLE_EXPERIMENTAL "Enable experimental features" OFF)so thestandalone device project accepts the forwarded flag and gates
add_subdirectory(experimental/ops).if(NOT ENABLE_EXPERIMENTAL) return()guards from 6 leaf CMakeLists underexperimental/ops(op_common,op_common/template,op_common/topo,reduce_scatter,reduce_scatter/birs,reduce_scatter/birs/template), per review. Gating stays single-sourced:the guard in
experimental/ops/CMakeLists.txtplus the twoadd_subdirectorycall sites(root
CMakeLists.txtfor host,cmake/device/CMakeLists.txtfor device); leaftarget_sources(hccl ...)remains protected by
if(TARGET hccl), so the device build (nohccltarget) is unaffected.experimental/ops/all_reduce/example:GetAlgNetMeta()signature adapted to the current base-classinterface (added
const OpParam& param).Related Issue
https://gitcode.com/cann/hccl/issues/324
https://gitcode.com/cann/hccl/issues/380
Testing
Testing with the hccl-checker tool via ST tests "/test/st/algorithm/testcase/reduce_scatter_testcase_a3.cc"

Single-node: ReduceScatterBirs was deployed on one Atlas A3 node with RankSize set between 4 and 16.
Example of ReduceScatterBIRS output from hccl_test for RankSize 16:
Documentation Update
docs/zh/build/build.mdanddocs/en/build/build.md: third-party software table, cann-cmake rowmaster-044 -> master-051. Download source switched from the cann-3rd OBS bucket
(
cmake-master-051.tar.gzis not published there — HTTP 403) to the raw.gitcode.com archive, whoseSHA256 matches the pin declared in
cmake/fetch_cann_cmake.cmake.Type Label