已合并
【2026 HCCL通信库创新大赛-华北赛区】【这对吗】 初赛及决赛代码归档 #900
【2026 HCCL通信库创新大赛-华北赛区】【这对吗】 初赛及决赛代码归档 #900
已合并
Celkryos创建于 8月2日
40 个文件变更+5629-0
@@ -1,3 +1,6 @@
1.idea/1.idea/
2.DS_Store2.DS_Store
3**/.DS_Store3**/.DS_Store
4+ 
5+# Windows download metadata copied by WSL/archive tools
6+*:Zone.Identifier
@@ -0,0 +1,106 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+Language: Cpp
12+Standard: c++17
13+TabWidth: 4
14+UseTab: Never
15+UseCRLF: false
16+AccessModifierOffset: -4
17+AlignConsecutiveAssignments: false
18+AlignConsecutiveDeclarations: false
19+AlignEscapedNewlines: DontAlign
20+AlignOperands: true
21+AlignTrailingComments: true
22+AllowAllArgumentsOnNextLine: true
23+AllowShortLambdasOnASingleLine: Empty
24+AllowAllParametersOfDeclarationOnNextLine: false
25+AllowShortBlocksOnASingleLine: false
26+AllowShortCaseLabelsOnASingleLine: false
27+AllowShortFunctionsOnASingleLine: false
28+AllowShortIfStatementsOnASingleLine: false
29+AllowShortLoopsOnASingleLine: false
30+AlwaysBreakAfterDefinitionReturnType: None
31+AlwaysBreakBeforeMultilineStrings: false
32+AlwaysBreakTemplateDeclarations: MultiLine
33+BinPackArguments: true
34+BinPackParameters: true
35+AlignAfterOpenBracket: DontAlign
36+ 
37+BraceWrapping:
38+ AfterCaseLabel: false
39+ AfterClass: false
40+ AfterControlStatement: Never
41+ AfterEnum: false
42+ AfterFunction: true
43+ AfterNamespace: false
44+ AfterStruct: false
45+ AfterUnion: false
46+ AfterExternBlock: false
47+ BeforeCatch: false
48+ BeforeElse: false
49+ IndentBraces: false
50+ SplitEmptyFunction: true
51+ SplitEmptyRecord: true
52+ SplitEmptyNamespace: true
53+ 
54+BreakBeforeBinaryOperators: All
55+BreakBeforeBraces: Custom
56+BreakBeforeTernaryOperators: true
57+BreakConstructorInitializersBeforeComma: false
58+BreakInheritanceList: AfterColon
59+ColumnLimit: 120
60+CommentPragmas: '^ IWYU pragma:'
61+PackConstructorInitializers: CurrentLine
62+ConstructorInitializerIndentWidth: 4
63+ContinuationIndentWidth: 4
64+Cpp11BracedListStyle: true
65+DerivePointerAlignment: false
66+DisableFormat: false
67+ExperimentalAutoDetectBinPacking: false
68+ForEachMacros: [ foreach ]
69+ 
70+IncludeCategories:
71+ - Regex: '^<'
72+ Priority: 3
73+ - Regex: '^"hccl'
74+ Priority: 2
75+ - Regex: '.*'
76+ Priority: 1
77+ 
78+IndentCaseLabels: true
79+IndentWidth: 4
80+IndentWrappedFunctionNames: false
81+KeepEmptyLinesAtTheStartOfBlocks: false
82+MacroBlockBegin: ''
83+MacroBlockEnd: ''
84+MaxEmptyLinesToKeep: 1
85+NamespaceIndentation: Inner
86+ 
87+PenaltyBreakBeforeFirstCallParameter: 19
88+PenaltyBreakComment: 300
89+PenaltyBreakFirstLessLess: 120
90+PenaltyBreakString: 1000
91+PenaltyExcessCharacter: 1000000
92+PenaltyReturnTypeOnItsOwnLine: 60
93+ 
94+PointerAlignment: Right
95+ReflowComments: true
96+SortIncludes: false
97+SpaceAfterCStyleCast: false
98+SpaceBeforeAssignmentOperators: true
99+SpaceBeforeParens: ControlStatements
100+SpaceInEmptyParentheses: false
101+SpacesBeforeTrailingComments: 1
102+SpacesInAngles: false
103+SpacesInContainerLiterals: true
104+SpacesInCStyleCastParentheses: false
105+SpacesInParentheses: false
106+SpacesInSquareBrackets: false
@@ -0,0 +1,11 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+* text=auto eol=lf
@@ -0,0 +1,35 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+# IDE
12+.vscode/
13+.idea/
14+.cache/
15+.DS_Store
16+*~
17+*.swp
18+*.swo
19+ 
20+# Build
21+build/
22+build_ut/
23+third_party/
24+__pycache__/
25+ 
26+# Agent
27+.omo/
28+.claude/
29+.opencode/
30+CLAUDE.md
31+AGENTS.md
32+ 
33+# Logs
34+logs/
35+*.log
@@ -0,0 +1,24 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+cmake_minimum_required(VERSION 3.16.0)
12+project(hccl_allgather)
13+ 
14+message(STATUS "CMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}")
15+message(STATUS "ASCEND_CANN_PACKAGE_PATH=${ASCEND_CANN_PACKAGE_PATH}")
16+ 
17+# 编译 Host 侧链接库
18+add_subdirectory(op_host)
19+add_subdirectory(op_kernel_ccu)
20+ 
21+# 安装
22+install(FILES include/hccl.h
23+ DESTINATION include
24+)
@@ -0,0 +1,201 @@
1+ Apache License
2+ Version 2.0, January 2004
3+ http://www.apache.org/licenses/
4+ 
5+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6+ 
7+ 1. Definitions.
8+ 
9+ "License" shall mean the terms and conditions for use, reproduction,
10+ and distribution as defined by Sections 1 through 9 of this document.
11+ 
12+ "Licensor" shall mean the copyright owner or entity authorized by
13+ the copyright owner that is granting the License.
14+ 
15+ "Legal Entity" shall mean the union of the acting entity and all
16+ other entities that control, are controlled by, or are under common
17+ control with that entity. For the purposes of this definition,
18+ "control" means (i) the power, direct or indirect, to cause the
19+ direction or management of such entity, whether by contract or
20+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21+ outstanding shares, or (iii) beneficial ownership of such entity.
22+ 
23+ "You" (or "Your") shall mean an individual or Legal Entity
24+ exercising permissions granted by this License.
25+ 
26+ "Source" form shall mean the preferred form for making modifications,
27+ including but not limited to software source code, documentation
28+ source, and configuration files.
29+ 
30+ "Object" form shall mean any form resulting from mechanical
31+ transformation or translation of a Source form, including but
32+ not limited to compiled object code, generated documentation,
33+ and conversions to other media types.
34+ 
35+ "Work" shall mean the work of authorship, whether in Source or
36+ Object form, made available under the License, as indicated by a
37+ copyright notice that is included in or attached to the work
38+ (an example is provided in the Appendix below).
39+ 
40+ "Derivative Works" shall mean any work, whether in Source or Object
41+ form, that is based on (or derived from) the Work and for which the
42+ editorial revisions, annotations, elaborations, or other modifications
43+ represent, as a whole, an original work of authorship. For the purposes
44+ of this License, Derivative Works shall not include works that remain
45+ separable from, or merely link (or bind by name) to the interfaces of,
46+ the Work and Derivative Works thereof.
47+ 
48+ "Contribution" shall mean any work of authorship, including
49+ the original version of the Work and any modifications or additions
50+ to that Work or Derivative Works thereof, that is intentionally
51+ submitted to Licensor for inclusion in the Work by the copyright owner
52+ or by an individual or Legal Entity authorized to submit on behalf of
53+ the copyright owner. For the purposes of this definition, "submitted"
54+ means any form of electronic, verbal, or written communication sent
55+ to the Licensor or its representatives, including but not limited to
56+ communication on electronic mailing lists, source code control systems,
57+ and issue tracking systems that are managed by, or on behalf of, the
58+ Licensor for the purpose of discussing and improving the Work, but
59+ excluding communication that is conspicuously marked or otherwise
60+ designated in writing by the copyright owner as "Not a Contribution."
61+ 
62+ "Contributor" shall mean Licensor and any individual or Legal Entity
63+ on behalf of whom a Contribution has been received by Licensor and
64+ subsequently incorporated within the Work.
65+ 
66+ 2. Grant of Copyright License. Subject to the terms and conditions of
67+ this License, each Contributor hereby grants to You a perpetual,
68+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69+ copyright license to reproduce, prepare Derivative Works of,
70+ publicly display, publicly perform, sublicense, and distribute the
71+ Work and such Derivative Works in Source or Object form.
72+ 
73+ 3. Grant of Patent License. Subject to the terms and conditions of
74+ this License, each Contributor hereby grants to You a perpetual,
75+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76+ (except as stated in this section) patent license to make, have made,
77+ use, offer to sell, sell, import, and otherwise transfer the Work,
78+ where such license applies only to those patent claims licensable
79+ by such Contributor that are necessarily infringed by their
80+ Contribution(s) alone or by combination of their Contribution(s)
81+ with the Work to which such Contribution(s) was submitted. If You
82+ institute patent litigation against any entity (including a
83+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84+ or a Contribution incorporated within the Work constitutes direct
85+ or contributory patent infringement, then any patent licenses
86+ granted to You under this License for that Work shall terminate
87+ as of the date such litigation is filed.
88+ 
89+ 4. Redistribution. You may reproduce and distribute copies of the
90+ Work or Derivative Works thereof in any medium, with or without
91+ modifications, and in Source or Object form, provided that You
92+ meet the following conditions:
93+ 
94+ (a) You must give any other recipients of the Work or
95+ Derivative Works a copy of this License; and
96+ 
97+ (b) You must cause any modified files to carry prominent notices
98+ stating that You changed the files; and
99+ 
100+ (c) You must retain, in the Source form of any Derivative Works
101+ that You distribute, all copyright, patent, trademark, and
102+ attribution notices from the Source form of the Work,
103+ excluding those notices that do not pertain to any part of
104+ the Derivative Works; and
105+ 
106+ (d) If the Work includes a "NOTICE" text file as part of its
107+ distribution, then any Derivative Works that You distribute must
108+ include a readable copy of the attribution notices contained
109+ within such NOTICE file, excluding those notices that do not
110+ pertain to any part of the Derivative Works, in at least one
111+ of the following places: within a NOTICE text file distributed
112+ as part of the Derivative Works; within the Source form or
113+ documentation, if provided along with the Derivative Works; or,
114+ within a display generated by the Derivative Works, if and
115+ wherever such third-party notices normally appear. The contents
116+ of the NOTICE file are for informational purposes only and
117+ do not modify the License. You may add Your own attribution
118+ notices within Derivative Works that You distribute, alongside
119+ or as an addendum to the NOTICE text from the Work, provided
120+ that such additional attribution notices cannot be construed
121+ as modifying the License.
122+ 
123+ You may add Your own copyright statement to Your modifications and
124+ may provide additional or different license terms and conditions
125+ for use, reproduction, or distribution of Your modifications, or
126+ for any such Derivative Works as a whole, provided Your use,
127+ reproduction, and distribution of the Work otherwise complies with
128+ the conditions stated in this License.
129+ 
130+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131+ any Contribution intentionally submitted for inclusion in the Work
132+ by You to the Licensor shall be under the terms and conditions of
133+ this License, without any additional terms or conditions.
134+ Notwithstanding the above, nothing herein shall supersede or modify
135+ the terms of any separate license agreement you may have executed
136+ with Licensor regarding such Contributions.
137+ 
138+ 6. Trademarks. This License does not grant permission to use the trade
139+ names, trademarks, service marks, or product names of the Licensor,
140+ except as required for reasonable and customary use in describing the
141+ origin of the Work and reproducing the content of the NOTICE file.
142+ 
143+ 7. Disclaimer of Warranty. Unless required by applicable law or
144+ agreed to in writing, Licensor provides the Work (and each
145+ Contributor provides its Contributions) on an "AS IS" BASIS,
146+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147+ implied, including, without limitation, any warranties or conditions
148+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149+ PARTICULAR PURPOSE. You are solely responsible for determining the
150+ appropriateness of using or redistributing the Work and assume any
151+ risks associated with Your exercise of permissions under this License.
152+ 
153+ 8. Limitation of Liability. In no event and under no legal theory,
154+ whether in tort (including negligence), contract, or otherwise,
155+ unless required by applicable law (such as deliberate and grossly
156+ negligent acts) or agreed to in writing, shall any Contributor be
157+ liable to You for damages, including any direct, indirect, special,
158+ incidental, or consequential damages of any character arising as a
159+ result of this License or out of the use or inability to use the
160+ Work (including but not limited to damages for loss of goodwill,
161+ work stoppage, computer failure or malfunction, or any and all
162+ other commercial damages or losses), even if such Contributor
163+ has been advised of the possibility of such damages.
164+ 
165+ 9. Accepting Warranty or Additional Liability. While redistributing
166+ the Work or Derivative Works thereof, You may choose to offer,
167+ and charge a fee for, acceptance of support, warranty, indemnity,
168+ or other liability obligations and/or rights consistent with this
169+ License. However, in accepting such obligations, You may act only
170+ on Your own behalf and on Your sole responsibility, not on behalf
171+ of any other Contributor, and only if You agree to indemnify,
172+ defend, and hold each Contributor harmless for any liability
173+ incurred by, or claims asserted against, such Contributor by reason
174+ of your accepting any such warranty or additional liability.
175+ 
176+ END OF TERMS AND CONDITIONS
177+ 
178+ APPENDIX: How to apply the Apache License to your work.
179+ 
180+ To apply the Apache License to your work, attach the following
181+ boilerplate notice, with the fields enclosed by brackets "{}"
182+ replaced with your own identifying information. (Don't include
183+ the brackets!) The text should be enclosed in the appropriate
184+ comment syntax for the file format. We also recommend that a
185+ file or class name and description of purpose be included on the
186+ same "printed page" as the copyright notice for easier
187+ identification within third-party archives.
188+ 
189+ Copyright {yyyy} {name of copyright owner}
190+ 
191+ Licensed under the Apache License, Version 2.0 (the "License");
192+ you may not use this file except in compliance with the License.
193+ You may obtain a copy of the License at
194+ 
195+ http://www.apache.org/licenses/LICENSE-2.0
196+ 
197+ Unless required by applicable law or agreed to in writing, software
198+ distributed under the License is distributed on an "AS IS" BASIS,
199+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200+ See the License for the specific language governing permissions and
201+ limitations under the License.
@@ -0,0 +1,64 @@
1+# AllGather 集合通信算子
2+ 
3+## 1. 项目介绍
4+ 
5+```
6+├── CMakeLists.txt # 顶层 CMake 配置
7+├── build.sh # 构建脚本
8+├── .clang-format # 代码风格配置
9+├── include/ # 头文件目录
10+│ ├── hccl.h # 集合通信算子头文件
11+│ ├── common.h # 通用数据结构定义
12+│ ├── custom.h # ★ 选手编写:自定义数据结构定义
13+│ ├── log.h # 日志宏定义
14+│ └── binary_stream.h # 序列化类定义
15+├── op_host/ # Host侧代码目录
16+│ ├── allgather.cc # ★ 选手编写:Host侧资源申请逻辑
17+│ └── exec_op.cc # ★ 选手编写:通信算法编排逻辑
18+└── op_kernel_ccu/ # CCU侧代码目录
19+ └── ccu_kernel.cc # ★ 选手编写:通信算法编排逻辑
20+```
21+ 
22+> [!NOTE] 注意:
23+> 算子工程中已提前预制好固有逻辑,选手仅允许修改 `custom.h`、`allgather.cc`、`exec_op.h`、`exec_op.cc`、`ccu_kernel.h`、`ccu_kernel.cc` 共 6 个文件内容。
24+ 
25+## 2. 编译运行
26+ 
27+### 2.1 安装 CANN-Toolkit 包
28+ 
29+请单击[下载链接](https://ascend.devcloud.huaweicloud.com/artifactory/cann-run-mirror/software/legacy/20260701000328953/),根据产品型号和环境架构下载对应软件包。安装命令如下,更多指导参考《[CANN软件安装指南](https://www.hiascend.com/document/redirect/CannCommunityInstWizard)》。
30+ 
31+```bash
32+# 确保安装包具有可执行权限
33+chmod +x Ascend-cann-toolkit_9.1.0_linux-${arch}.run
34+# 安装命令
35+./Ascend-cann-toolkit_9.1.0_linux-${arch}.run --full --install-path=${install_path}
36+```
37+ 
38+### 2.2 环境变量配置
39+ 
40+按需选择合适的命令使环境变量生效。
41+ 
42+```bash
43+# 默认路径安装,以root用户为例(非root用户,将/usr/local替换为${HOME})
44+source /usr/local/Ascend/cann/set_env.sh
45+# 指定路径安装
46+# source ${install_path}/cann/set_env.sh
47+```
48+ 
49+### 2.3 编译算子工程
50+ 
51+```bash
52+bash build.sh
53+ 
54+# 编译 Debug 版本,便于断点调试
55+bash build.sh --debug
56+```
57+ 
58+## 3. 代码格式
59+ 
60+选手代码需符合 [.clang-format](.clang-format) 文件中的代码风格规范,可通过下列命令一键修改:
61+ 
62+```bash
63+bash build.sh --format
64+```
@@ -0,0 +1,101 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+set -e
12+ 
13+PROJECT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
14+BUILD_DIR="${PROJECT_DIR}/build"
15+BUILD_TYPE="Release"
16+ENABLE_FORMAT="OFF"
17+ 
18+CPU_NUM="$(nproc)"
19+ASCEND_CANN_PACKAGE_PATH="/usr/local/Ascend/cann"
20+ 
21+usage() {
22+ cat <<EOF
23+Usage: $0 [OPTIONS]
24+ 
25+Options:
26+ --debug 编译 Debug 版本
27+ --format 格式化代码
28+ -h, --help 显示帮助信息
29+EOF
30+ exit 0
31+}
32+ 
33+parse_args() {
34+ local opts
35+ opts=$(getopt -o h -l debug,format,help -- "$@") || usage
36+ eval set -- "${opts}"
37+ 
38+ for arg; do
39+ case "${arg}" in
40+ --debug) BUILD_TYPE="Debug" ;;
41+ --format) ENABLE_FORMAT="ON" ;;
42+ -h|--help) usage ;;
43+ esac
44+ done
45+}
46+ 
47+parse_cann_path() {
48+ if [[ -z "${ASCEND_HOME_PATH}" ]]; then
49+ printf "ERROR: ASCEND_HOME_PATH is not set.\n" >&2
50+ printf "Please ensure CANN-Toolkit is properly installed and source environment variables by running:\n" >&2
51+ printf " source /path/to/Ascend/cann/set_env.sh\n" >&2
52+ exit 1
53+ fi
54+ 
55+ # 使用 local 避免污染全局(如需要全局可去掉 local)
56+ ASCEND_CANN_PACKAGE_PATH="${ASCEND_HOME_PATH}"
57+ return 0
58+}
59+ 
60+build() {
61+ # 创建构建目录
62+ cd "${PROJECT_DIR}"
63+ mkdir -p "${BUILD_DIR}"
64+ 
65+ # 配置
66+ cmake -S . -B "${BUILD_DIR}" \
67+ -DCMAKE_EXPORT_COMPILE_COMMANDS=ON \
68+ -DCMAKE_BUILD_TYPE="${BUILD_TYPE}" \
69+ -DCMAKE_INSTALL_PREFIX="${BUILD_DIR}" \
70+ -DASCEND_CANN_PACKAGE_PATH="${ASCEND_CANN_PACKAGE_PATH}"
71+ 
72+ # 编译
73+ cmake --build "${BUILD_DIR}" -j${CPU_NUM}
74+ 
75+ # 安装
76+ cmake --install "${BUILD_DIR}"
77+}
78+ 
79+format() {
80+ cd "${PROJECT_DIR}"
81+ find ./include -type f -regex '.*\.\(h\|hpp\|cpp\|cc\)$' -exec clang-format -i {} \;
82+ find ./op_host -type f -regex '.*\.\(h\|hpp\|cpp\|cc\)$' -exec clang-format -i {} \;
83+ find ./op_kernel_ccu -type f -regex '.*\.\(h\|hpp\|cpp\|cc\)$' -exec clang-format -i {} \;
84+}
85+ 
86+main() {
87+ # 解析参数
88+ parse_args "$@"
89+ # 解析 CANN-Toolkit 路径
90+ parse_cann_path
91+ 
92+ if [[ "${ENABLE_FORMAT}" == "ON" ]]; then
93+ # 格式化代码
94+ format
95+ else
96+ # 编译
97+ build
98+ fi
99+}
100+ 
101+main "$@"
@@ -0,0 +1,168 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_BINARY_STREAM_H
12+#define OPS_HCCL_BINARY_STREAM_H
13+ 
14+#include <vector>
15+#include <cstdint>
16+#include <sstream>
17+#include <algorithm>
18+#include <map>
19+ 
20+#include "log.h"
21+ 
22+/**
23+ * @brief 序列化工具类
24+ */
25+class BinaryStream {
26+public:
27+ static constexpr std::ios_base::openmode DEFAULT_IOS_MODE = std::ios_base::in | std::ios_base::out;
28+ 
29+ explicit BinaryStream(std::ios_base::openmode mode = DEFAULT_IOS_MODE) : stream(mode | std::ios_base::binary) {};
30+ 
31+ explicit BinaryStream(std::vector<char> &buf, std::ios_base::openmode mode = DEFAULT_IOS_MODE)
32+ : stream(mode | std::ios_base::binary)
33+ {
34+ stream.rdbuf()->pubsetbuf(buf.data(), buf.size());
35+ }
36+ 
37+ template <typename T> BinaryStream &operator<<(const T &t)
38+ {
39+ stream.write(reinterpret_cast<const char *>(&t), sizeof(T));
40+ return *this;
41+ }
42+ 
43+ // 多级vector递归序列化
44+ template <typename T> BinaryStream &operator<<(const std::vector<T> &vec)
45+ {
46+ size_t size = vec.size();
47+ *this << size;
48+ for (const auto &elem : vec) {
49+ *this << elem;
50+ }
51+ return *this;
52+ }
53+ 
54+ // 对string的输入函数
55+ BinaryStream &operator<<(const std::string &s)
56+ {
57+ size_t size = s.size();
58+ stream.write(reinterpret_cast<const char *>(&size), sizeof(size_t)); // 写入长度
59+ stream.write(s.data(), size); // 写入字符数据
60+ return *this;
61+ }
62+ 
63+ template <typename T> BinaryStream &operator>>(T &t)
64+ {
65+ stream.read(reinterpret_cast<char *>(&t), sizeof(T));
66+ return *this;
67+ }
68+ 
69+ // 对string的读取函数
70+ BinaryStream &operator>>(std::string &s)
71+ {
72+ size_t size;
73+ stream.read(reinterpret_cast<char *>(&size), sizeof(size)); // 先从流中读取字符串长度
74+ s.resize(size); // 为string分配足够空间
75+ stream.read(&s[0], size); // 直接读取数据到string的缓冲区中,无需再分配内存
76+ return *this;
77+ }
78+ 
79+ // 多级vector递归反序列化
80+ template <typename T> BinaryStream &operator>>(std::vector<T> &vec)
81+ {
82+ size_t size;
83+ *this >> size;
84+ vec.resize(size);
85+ for (auto &elem : vec) {
86+ *this >> elem;
87+ }
88+ return *this;
89+ }
90+ 
91+ // map序列化
92+ template <typename T1, typename T2> BinaryStream &operator<<(const std::map<T1, T2> &m)
93+ {
94+ size_t size = m.size();
95+ *this << size;
96+ for (const auto &elem : m) {
97+ *this << elem.first;
98+ *this << elem.second;
99+ }
100+ return *this;
101+ }
102+ 
103+ // map反序列化
104+ template <typename T1, typename T2> BinaryStream &operator>>(std::map<T1, T2> &m)
105+ {
106+ size_t size;
107+ *this >> size;
108+ for (size_t i = 0; i < size; i++) {
109+ T1 key;
110+ *this >> key;
111+ T2 value;
112+ *this >> value;
113+ m[key] = value;
114+ }
115+ return *this;
116+ }
117+ 
118+ void Dump(std::vector<char> &vec)
119+ {
120+ std::for_each(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>(), [&vec](const char c) {
121+ vec.push_back(c);
122+ });
123+ }
124+ 
125+ void DumpWithRevert(std::vector<char> &vec)
126+ {
127+ std::streampos originalPos = stream.tellg(); // 保存原始位置
128+ std::for_each(std::istreambuf_iterator<char>(stream), std::istreambuf_iterator<char>(), [&vec](const char c) {
129+ vec.push_back(c);
130+ });
131+ stream.seekg(originalPos); // 恢复原始位置
132+ }
133+ 
134+ std::uint64_t GetSize()
135+ {
136+ return stream.str().size();
137+ }
138+ 
139+ std::string GetString()
140+ {
141+ return stream.str();
142+ }
143+ 
144+ std::string SplictStream(uint64_t &start, uint64_t &end)
145+ {
146+ std::string temp = stream.str();
147+ if (start >= temp.length()) {
148+ HCCL_ERROR("[SplictStream]start[%lu] is bigger than stream length[%lu]", start, temp.length());
149+ return "";
150+ }
151+ 
152+ // 截取子串
153+ std::string result = temp.substr(start, end - start);
154+ 
155+ // 返回新的 string
156+ return result;
157+ }
158+ 
159+ void Clear()
160+ {
161+ stream.clear();
162+ }
163+ 
164+private:
165+ std::stringstream stream;
166+};
167+ 
168+#endif // OPS_HCCL_BINARY_STREAM_H
@@ -0,0 +1,54 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_COMMON_H
12+#define OPS_HCCL_COMMON_H
13+ 
14+#include <unordered_map>
15+ 
16+#include <hccl/hccl_types.h>
17+#include <hccl/hccl_res.h>
18+#include <hccl/hcomm_primitives.h>
19+#include <ccu/ccu_types.h>
20+#include <ccu/ccu_variable.hpp>
21+#include <ccu/ccu_event.hpp>
22+#include <ccu/ccu_primitives.hpp>
23+#include <acl/acl_rt.h>
24+ 
25+constexpr uint32_t NOTIFY_IDX_ACK = 0;
26+constexpr uint32_t NOTIFY_IDX_DATA_SIGNAL = 1;
27+constexpr uint32_t CUSTOM_TIMEOUT = 1800;
28+ 
29+constexpr uint32_t COMM_INDENTIFIER_MAX_LENGTH = 128;
30+constexpr uint32_t OP_NAME_LENGTH = 32;
31+constexpr uint32_t TAG_LENGTH = OP_NAME_LENGTH + COMM_INDENTIFIER_MAX_LENGTH;
32+constexpr uint32_t INVALID_VALUE_RANKID = 0xFFFFFFFF;
33+constexpr uint32_t MAX_DATA_SIZE = 256 * 1024 * 1024; // 单次通信的最大数据量,256MB
34+constexpr uint64_t MAX_RANK_SIZE = 16;
35+ 
36+struct OpParam {
37+ char tag[TAG_LENGTH];
38+ void *inputPtr = nullptr;
39+ void *outputPtr = nullptr;
40+ uint64_t count = 0;
41+ uint32_t root = 0;
42+ uint32_t myRank = INVALID_VALUE_RANKID;
43+ uint32_t rankSize = 0;
44+ HcclDataType dataType = HCCL_DATA_TYPE_RESERVED;
45+ HcclCMDType opType = HcclCMDType::HCCL_CMD_INVALID;
46+ HcclReduceOp reduceType = HcclReduceOp::HCCL_REDUCE_SUM;
47+ ThreadHandle cpuThread;
48+ void *resCtx = nullptr; ///< 通信引擎上下文中的资源信息,存放 AlgResourceCtx 序列化后的内容
49+ uint64_t ctxSize = 0;
50+};
51+ 
52+const std::unordered_map<HcclDataType, uint32_t> SIZE_TABLE = {{HCCL_DATA_TYPE_FP32, sizeof(float)}};
53+ 
54+#endif // OPS_HCCL_COMMON_H
@@ -0,0 +1,113 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_CUSTOM_H
12+#define OPS_HCCL_CUSTOM_H
13+ 
14+#include <memory>
15+#include <hccl/hccl_types.h>
16+#include <hccl/hccl_res.h>
17+ 
18+#include "binary_stream.h"
19+#include "common.h"
20+ 
21+typedef struct {
22+ void *addr;
23+ uint64_t size;
24+} CommBuffer;
25+ 
26+// 传输模式(taskArgs[6]):扁平直推 / 三路拆分阶段一 / 三路拆分阶段二
27+constexpr uint64_t TRANSFER_FLAT = 0;
28+constexpr uint64_t TRANSFER_PHASE_ONE = 1;
29+constexpr uint64_t TRANSFER_PHASE_TWO = 2;
30+// 4*1 大消息递归倍增(单 launch 融合两轮):round1 与 r^1 交换 own S,
31+// 内部 barrier 后 round2 与 r^2 交换相邻两片(2S);每轮单 channel 独占全 NIC
32+constexpr uint64_t TRANSFER_ROUND_FUSED = 3;
33+ 
34+struct CcuKernelArgBase {
35+ ChannelHandle channels[MAX_RANK_SIZE]{};
36+ uint32_t channelCount = 0;
37+};
38+ 
39+// 每个 Kernel 只持有同一网络层、同一 IO Die 上的 Channel。
40+// copySelf 用于把本 Rank 的本地拷贝分配给负载较轻的 Kernel。
41+struct CcuKernelArgAllGather : public CcuKernelArgBase {
42+ uint32_t copySelf = 0;
43+ uint32_t isLocalLayer = 0;
44+ uint32_t hierarchicalEnabled = 0;
45+ uint32_t myRank = 0;
46+ uint32_t pairRank = 0;
47+ uint32_t pairChannelIdx = 0;
48+ uint32_t pairChannelIdxR2 = 0; // 4*1 递归倍增 round-2 的配对 channel(r^2)
49+ uint32_t localRankCount = 0;
50+ uint32_t remoteRankCount = 0;
51+ uint32_t localRanks[MAX_RANK_SIZE]{};
52+ uint32_t ownedRemoteCount = 0;
53+ uint32_t ownedRemoteRanks[MAX_RANK_SIZE]{};
54+ // owned-remote 在 layer-1 group 内的 channel 下标(与 ownedRemoteRanks 一一对应)
55+ uint32_t ownedRemoteChannelIdx[MAX_RANK_SIZE]{};
56+ uint32_t stageTwoTargetCount = 0;
57+ uint32_t stageTwoTargetChannelIdx[MAX_RANK_SIZE]{};
58+};
59+ 
60+// ccu kernel register所需信息
61+struct CcuKernelInfo {
62+ // kernel名称
63+ char kernelFuncName[64]{};
64+ // kernel函数
65+ void *kernelFunc = nullptr;
66+ // KernelArg实例指针
67+ void *kernelArg = nullptr;
68+ 
69+private:
70+ std::shared_ptr<CcuKernelArgBase> kernelArgSmartPtr;
71+ 
72+public:
73+ template <typename T> void setKernelArg(std::shared_ptr<T> arg)
74+ {
75+ kernelArgSmartPtr = std::static_pointer_cast<CcuKernelArgBase>(arg);
76+ kernelArg = static_cast<void *>(arg.get());
77+ }
78+};
79+ 
80+struct AlgResourceCtx {
81+ ThreadHandle ccuThread{}; ///< CCU通信引擎上的thread资源
82+ CommBuffer localBuffer{}; ///< 本端HCCL通信内存
83+ std::vector<ThreadHandle> threads; ///< CCU通信引擎上的thread资源
84+ std::vector<CcuKernelHandle> ccuKernels; ///< 分层 kernel(与 threads 一一对应)
85+ std::vector<CcuKernelHandle> flatKernels; ///< 极简扁平 kernel(与 threads 一一对应)
86+ 
87+ // 序列化
88+ std::vector<char> Serialize()
89+ {
90+ BinaryStream binaryStream;
91+ binaryStream << ccuThread;
92+ binaryStream << localBuffer;
93+ binaryStream << threads;
94+ binaryStream << ccuKernels;
95+ binaryStream << flatKernels;
96+ std::vector<char> result;
97+ binaryStream.Dump(result);
98+ return result;
99+ }
100+ 
101+ // 反序列化
102+ void DeSerialize(std::vector<char> &data)
103+ {
104+ BinaryStream binaryStream(data);
105+ binaryStream >> ccuThread;
106+ binaryStream >> localBuffer;
107+ binaryStream >> threads;
108+ binaryStream >> ccuKernels;
109+ binaryStream >> flatKernels;
110+ }
111+};
112+ 
113+#endif // OPS_HCCL_CUSTOM_H
@@ -0,0 +1,31 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_H
12+#define OPS_HCCL_H
13+ 
14+#include <acl/acl.h>
15+#include <hccl/hccl_comm.h>
16+#include <hccl/hccl_res.h>
17+#include <hccl/hccl_types.h>
18+ 
19+#ifdef __cplusplus
20+extern "C" {
21+#endif
22+ 
23+/* AllGather 算子入口 */
24+HcclResult HcclAllGather(
25+ void *sendBuf, void *recvBuf, uint64_t sendCount, HcclDataType dataType, HcclComm comm, aclrtStream stream);
26+ 
27+#ifdef __cplusplus
28+}
29+#endif
30+ 
31+#endif // OPS_HCCL_H
@@ -0,0 +1,140 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_LOG_H
12+#define OPS_HCCL_LOG_H
13+ 
14+#include <cstdio>
15+#include <hccl/hccl_types.h>
16+#include <ccu/ccu_types.h>
17+ 
18+#ifndef LOG_LEVEL
19+#define LOG_LEVEL LOG_LEVEL_ERROR // 默认日志级别为:ERROR
20+#endif
21+ 
22+typedef enum {
23+ LOG_LEVEL_DEBUG = 0, // DEBUG级别
24+ LOG_LEVEL_INFO = 1, // INFO级别
25+ LOG_LEVEL_WARNING = 2, // WARNING级别
26+ LOG_LEVEL_ERROR = 3, // ERROR级别
27+ LOG_LEVEL_NONE = 4 // 关闭所有日志
28+} LogLevel;
29+ 
30+// CcuResult 返回码转换为 HcclResult 返回码
31+inline HcclResult ConvertCcuToHccl(CcuResult ccuResult)
32+{
33+ switch (ccuResult) {
34+ case CCU_SUCCESS:
35+ return HCCL_SUCCESS;
36+ case CCU_E_PARA:
37+ return HCCL_E_PARA;
38+ case CCU_E_PTR:
39+ return HCCL_E_PTR;
40+ case CCU_E_INTERNAL:
41+ return HCCL_E_INTERNAL;
42+ case CCU_E_NOT_SUPPORT:
43+ return HCCL_E_NOT_SUPPORT;
44+ case CCU_E_NOT_FOUND:
45+ return HCCL_E_NOT_FOUND;
46+ case CCU_E_UNAVAIL:
47+ return HCCL_E_UNAVAIL;
48+ default:
49+ return HCCL_E_INTERNAL;
50+ }
51+}
52+ 
53+#ifndef LIKELY
54+#define LIKELY(x) (static_cast<bool>(__builtin_expect(static_cast<bool>(x), 1)))
55+#define UNLIKELY(x) (static_cast<bool>(__builtin_expect(static_cast<bool>(x), 0)))
56+#endif
57+ 
58+#define HCCL_DEBUG(format, ...) \
59+ do { \
60+ if (LOG_LEVEL <= LOG_LEVEL_DEBUG) { \
61+ printf("[DEBUG][%s][%s:%d]" format "\n", __func__, __FILE__, __LINE__, ##__VA_ARGS__); \
62+ } \
63+ } while (0)
64+ 
65+#define HCCL_INFO(format, ...) \
66+ do { \
67+ if (LOG_LEVEL <= LOG_LEVEL_INFO) { \
68+ printf("[INFO][%s][%s:%d]" format "\n", __func__, __FILE__, __LINE__, ##__VA_ARGS__); \
69+ } \
70+ } while (0)
71+ 
72+#define HCCL_WARNING(format, ...) \
73+ do { \
74+ if (LOG_LEVEL <= LOG_LEVEL_WARNING) { \
75+ printf("[WARN][%s][%s:%d]" format "\n", __func__, __FILE__, __LINE__, ##__VA_ARGS__); \
76+ } \
77+ } while (0)
78+ 
79+#define HCCL_ERROR(format, ...) \
80+ do { \
81+ if (LOG_LEVEL <= LOG_LEVEL_ERROR) { \
82+ printf("[ERROR][%s][%s:%d]" format "\n", __func__, __FILE__, __LINE__, ##__VA_ARGS__); \
83+ } \
84+ } while (0)
85+ 
86+/* 检查指针, 若指针为NULL, 则记录日志, 并返回错误 */
87+#define CHK_PTR_NULL(ptr) \
88+ do { \
89+ if (UNLIKELY((ptr) == nullptr)) { \
90+ HCCL_ERROR("[%s] ptr [%s] is nullptr, return HCCL_E_PTR", __func__, #ptr); \
91+ return HCCL_E_PTR; \
92+ } \
93+ } while (0)
94+ 
95+/* 检查函数返回值, 记录指定日志, 并返回指定错误码 */
96+#define CHK_PRT_RET(result, exeLog, retCode) \
97+ do { \
98+ if (UNLIKELY(result)) { \
99+ exeLog; \
100+ return retCode; \
101+ } \
102+ } while (0)
103+ 
104+/* 检查函数返回值, 并返回指定错误码 */
105+#define CHK_RET(call) \
106+ do { \
107+ int32_t hcclRet = call; \
108+ if (UNLIKELY(hcclRet != HCCL_SUCCESS)) { \
109+ if (hcclRet == HCCL_E_AGAIN) { \
110+ HCCL_WARNING("[%s] call trace: hcclRet -> %d", __func__, hcclRet); \
111+ } else { \
112+ HCCL_ERROR("[%s] call trace: hcclRet -> %d", __func__, hcclRet); \
113+ } \
114+ return static_cast<HcclResult>(hcclRet); \
115+ } \
116+ } while (0)
117+ 
118+/* 检查函数返回值, 并返回指定错误码 */
119+#define CHK_RET_CCU(call) \
120+ do { \
121+ CcuResult ccuRet = call; \
122+ if (UNLIKELY(ccuRet != CCU_SUCCESS)) { \
123+ HCCL_ERROR("[%s] call trace: ccuRet -> %d", __func__, ccuRet); \
124+ return ConvertCcuToHccl(ccuRet); \
125+ } \
126+ } while (0)
127+ 
128+#define ACLCHECK(cmd) \
129+ do { \
130+ aclError ret = cmd; \
131+ if (UNLIKELY(ret != ACL_SUCCESS)) { \
132+ HCCL_ERROR("acl interface return err %s:%d, retcode: %d.\n", __FILE__, __LINE__, ret); \
133+ if (ret == ACL_ERROR_RT_MEMORY_ALLOCATION) { \
134+ HCCL_ERROR("memory allocation error, check whether the current memory space is sufficient.\n"); \
135+ } \
136+ return HCCL_E_RUNTIME; \
137+ } \
138+ } while (0)
139+ 
140+#endif // OPS_HCCL_LOG_H
@@ -0,0 +1,69 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+# ==================================================
12+# Host 侧动态链接库
13+# ==================================================
14+add_library(hccl SHARED)
15+ 
16+target_sources(hccl PRIVATE
17+ ${CMAKE_CURRENT_SOURCE_DIR}/allgather.cc
18+ ${CMAKE_CURRENT_SOURCE_DIR}/exec_op.cc
19+)
20+ 
21+target_include_directories(hccl PRIVATE
22+ ${CMAKE_CURRENT_SOURCE_DIR}/
23+ ${CMAKE_CURRENT_SOURCE_DIR}/../include
24+ # CANN Toolkit头文件
25+ ${ASCEND_CANN_PACKAGE_PATH}/include
26+ ${ASCEND_CANN_PACKAGE_PATH}/include/hccl
27+ ${ASCEND_CANN_PACKAGE_PATH}/include/hcomm
28+ ${ASCEND_CANN_PACKAGE_PATH}/include/hcomm/ccu
29+ ${ASCEND_CANN_PACKAGE_PATH}/pkg_inc
30+ ${ASCEND_CANN_PACKAGE_PATH}/pkg_inc/hccl
31+ ${ASCEND_CANN_PACKAGE_PATH}/pkg_inc/hcomm
32+ ${ASCEND_CANN_PACKAGE_PATH}/pkg_inc/hcomm/ccu
33+)
34+ 
35+target_compile_options(hccl PRIVATE
36+ -std=c++17
37+ -Werror
38+ -Wall
39+ -fno-common
40+ -fno-strict-aliasing
41+ -pipe
42+ -fstack-protector-all
43+ -U_FORTIFY_SOURCE
44+ $<$<CONFIG:Debug>:-g -O0>
45+ $<$<CONFIG:Release>:-O2>
46+)
47+ 
48+target_compile_definitions(hccl PRIVATE
49+ _GLIBCXX_USE_CXX11_ABI=0
50+)
51+ 
52+target_link_options(hccl PRIVATE
53+ -Wl,-z,relro
54+ -Wl,-z,now
55+ -Wl,-z,noexecstack
56+)
57+ 
58+target_link_directories(hccl PRIVATE
59+ ${ASCEND_CANN_PACKAGE_PATH}/lib64
60+)
61+ 
62+target_link_libraries(hccl PRIVATE
63+ hcomm
64+ acl_rt
65+)
66+ 
67+install(TARGETS hccl LIBRARY
68+ DESTINATION lib64
69+)
@@ -0,0 +1,493 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#include <algorithm>
12+#include <cstdio>
13+#include <map>
14+#include <memory>
15+#include <mutex>
16+#include <utility>
17+#include <vector>
18+ 
19+#include <ccu/ccu_launch.h>
20+#include <hccl/hccl_ccu_res.h>
21+#include <hccl/hccl_diag.h>
22+#include <hccl/hccl_rank_graph.h>
23+#include <hccl/hccl_res_expt.h>
24+ 
25+#include "common.h"
26+#include "custom.h"
27+#include "ccu_kernel.h"
28+#include "exec_op.h"
29+#include "hccl.h"
30+#include "log.h"
31+ 
32+namespace {
33+ 
34+constexpr uint32_t CHANNEL_NOTIFY_NUM = 3;
35+constexpr uint32_t MAX_KERNEL_NUM = 2;
36+constexpr uint32_t KERNEL_ARG_NUM = 1;
37+constexpr uint32_t RESERVED_DIE_ID = 0;
38+ 
39+struct ChannelPlan {
40+ HcclChannelDesc desc{};
41+ uint32_t remoteRank = INVALID_VALUE_RANKID;
42+ uint32_t netLayer = 0;
43+ EndpointAttrDieId dieId = 0;
44+};
45+ 
46+struct ChannelGroup {
47+ uint32_t netLayer = 0;
48+ EndpointAttrDieId dieId = 0;
49+ std::vector<ChannelPlan> plans;
50+ std::vector<ChannelHandle> handles;
51+};
52+ 
53+HcclResult ConvertCcuResult(CcuResult result)
54+{
55+ switch (result) {
56+ case CCU_SUCCESS:
57+ return HCCL_SUCCESS;
58+ case CCU_E_PARA:
59+ return HCCL_E_PARA;
60+ case CCU_E_PTR:
61+ return HCCL_E_PTR;
62+ case CCU_E_INTERNAL:
63+ return HCCL_E_INTERNAL;
64+ case CCU_E_NOT_SUPPORT:
65+ return HCCL_E_NOT_SUPPORT;
66+ case CCU_E_NOT_FOUND:
67+ return HCCL_E_NOT_FOUND;
68+ case CCU_E_UNAVAIL:
69+ return HCCL_E_UNAVAIL;
70+ default:
71+ return HCCL_E_INTERNAL;
72+ }
73+}
74+ 
75+HcclResult SelectLink(
76+ HcclComm comm, const OpParam &param, uint32_t remoteRank, const std::vector<uint32_t> &netLayers, ChannelPlan &plan)
77+{
78+ for (uint32_t netLayer : netLayers) {
79+ CommLink *links = nullptr;
80+ uint32_t linkNum = 0;
81+ CHK_RET(HcclRankGraphGetLinks(comm, netLayer, param.myRank, remoteRank, &links, &linkNum));
82+ 
83+ for (uint32_t linkIdx = 0; linkIdx < linkNum; ++linkIdx) {
84+ const CommLink link = links[linkIdx];
85+ if (link.linkAttr.linkProtocol != CommProtocol::COMM_PROTOCOL_UBC_CTP) {
86+ continue;
87+ }
88+ 
89+ CHK_RET(HcclChannelDescInit(&plan.desc, 1));
90+ plan.desc.remoteRank = remoteRank;
91+ plan.desc.notifyNum = CHANNEL_NOTIFY_NUM;
92+ plan.desc.channelProtocol = link.linkAttr.linkProtocol;
93+ plan.desc.localEndpoint = link.srcEndpointDesc;
94+ plan.desc.remoteEndpoint = link.dstEndpointDesc;
95+ plan.remoteRank = remoteRank;
96+ plan.netLayer = netLayer;
97+ 
98+ CHK_RET(HcclRankGraphGetEndpointInfo(
99+ comm, param.myRank, &plan.desc.localEndpoint, ENDPOINT_ATTR_DIE_ID, sizeof(plan.dieId), &plan.dieId));
100+ return HCCL_SUCCESS;
101+ }
102+ }
103+ 
104+ HCCL_ERROR("[AllGather] No UBC_CTP link from rank %u to rank %u", param.myRank, remoteRank);
105+ return HCCL_E_NOT_FOUND;
106+}
107+ 
108+HcclResult BuildChannelGroups(HcclComm comm, const OpParam &param, std::vector<ChannelGroup> &groups)
109+{
110+ uint32_t *netLayerList = nullptr;
111+ uint32_t netLayerNum = 0;
112+ CHK_RET(HcclRankGraphGetLayers(comm, &netLayerList, &netLayerNum));
113+ CHK_PRT_RET(netLayerList == nullptr || netLayerNum == 0, HCCL_ERROR("[AllGather] Rank graph has no network layer"),
114+ HCCL_E_INTERNAL);
115+ 
116+ std::vector<uint32_t> netLayers(netLayerList, netLayerList + netLayerNum);
117+ std::sort(netLayers.begin(), netLayers.end());
118+ 
119+ std::map<std::pair<uint32_t, EndpointAttrDieId>, std::vector<ChannelPlan>> groupedPlans;
120+ for (uint32_t remoteRank = 0; remoteRank < param.rankSize; ++remoteRank) {
121+ if (remoteRank == param.myRank) {
122+ continue;
123+ }
124+ 
125+ ChannelPlan plan;
126+ CHK_RET(SelectLink(comm, param, remoteRank, netLayers, plan));
127+ groupedPlans[{plan.netLayer, plan.dieId}].push_back(plan);
128+ }
129+ 
130+ CHK_PRT_RET(groupedPlans.empty(),
131+ HCCL_ERROR("[AllGather] No channel group was generated for rank %u", param.myRank), HCCL_E_INTERNAL);
132+ CHK_PRT_RET(groupedPlans.size() > MAX_KERNEL_NUM,
133+ HCCL_ERROR("[AllGather] Need %zu CCU kernels, but final topology supports at most %u", groupedPlans.size(),
134+ MAX_KERNEL_NUM),
135+ HCCL_E_NOT_SUPPORT);
136+ 
137+ groups.reserve(groupedPlans.size());
138+ for (auto &entry : groupedPlans) {
139+ ChannelGroup group;
140+ group.netLayer = entry.first.first;
141+ group.dieId = entry.first.second;
142+ group.plans = std::move(entry.second);
143+ 
144+ std::vector<HcclChannelDesc> descs;
145+ descs.reserve(group.plans.size());
146+ for (const ChannelPlan &plan : group.plans) {
147+ descs.push_back(plan.desc);
148+ }
149+ 
150+ group.handles.resize(descs.size());
151+ CHK_RET(HcclChannelAcquire(comm, CommEngine::COMM_ENGINE_CCU, descs.data(), static_cast<uint32_t>(descs.size()),
152+ group.handles.data()));
153+ HCCL_INFO("[AllGather] rank %u: layer %u, die %u, channel count %zu", param.myRank, group.netLayer, group.dieId,
154+ group.handles.size());
155+ groups.push_back(std::move(group));
156+ }
157+ return HCCL_SUCCESS;
158+}
159+ 
160+HcclResult RegisterKernels(
161+ HcclComm comm, const OpParam &param, const std::vector<ChannelGroup> &groups, AlgResourceCtx &resCtx)
162+{
163+ CcuInsHandle insHandle{0};
164+ uint32_t insNum = 0;
165+ CHK_RET(HcclCommQueryCcuIns(comm, &insHandle, &insNum));
166+ CHK_PRT_RET(insNum != 1, HCCL_ERROR("[AllGather] Expected one CCU instance, got %u", insNum), HCCL_E_INTERNAL);
167+ 
168+ size_t copyKernelIdx = 0;
169+ for (size_t idx = 1; idx < groups.size(); ++idx) {
170+ if (groups[idx].handles.size() < groups[copyKernelIdx].handles.size()) {
171+ copyKernelIdx = idx;
172+ }
173+ }
174+ 
175+ size_t localGroupIdx = groups.size();
176+ size_t remoteGroupIdx = groups.size();
177+ std::vector<uint32_t> localRanks{param.myRank};
178+ std::vector<uint32_t> remoteRanks;
179+ for (size_t idx = 0; idx < groups.size(); ++idx) {
180+ if (groups[idx].netLayer == 0) {
181+ localGroupIdx = idx;
182+ for (const ChannelPlan &plan : groups[idx].plans) {
183+ localRanks.push_back(plan.remoteRank);
184+ }
185+ } else if (groups[idx].netLayer == 1) {
186+ remoteGroupIdx = idx;
187+ for (const ChannelPlan &plan : groups[idx].plans) {
188+ remoteRanks.push_back(plan.remoteRank);
189+ }
190+ }
191+ }
192+ std::sort(localRanks.begin(), localRanks.end());
193+ std::sort(remoteRanks.begin(), remoteRanks.end());
194+ 
195+ const bool hasTwoLayers = localGroupIdx < groups.size() && remoteGroupIdx < groups.size();
196+ const bool isTwoByEight = param.rankSize == 16 && hasTwoLayers && localRanks.size() == 8 && remoteRanks.size() == 8;
197+ const bool isEightPlusFour = param.rankSize == 12 && hasTwoLayers
198+ && ((localRanks.size() == 8 && remoteRanks.size() == 4)
199+ || (localRanks.size() == 4 && remoteRanks.size() == 8));
200+ const bool hierarchicalEnabled = isTwoByEight || isEightPlusFour;
201+ uint32_t pairRank = param.myRank;
202+ uint32_t pairChannelIdx = 0;
203+ std::vector<uint32_t> ownedRemoteRanks;
204+ std::vector<uint32_t> ownedRemoteChannelIdx;
205+ std::vector<uint32_t> stageTwoTargetChannelIdx;
206+ if (hierarchicalEnabled) {
207+ const auto localRankIt = std::find(localRanks.begin(), localRanks.end(), param.myRank);
208+ CHK_PRT_RET(localRankIt == localRanks.end(),
209+ HCCL_ERROR("[AllGather] rank %u is absent from its local rank group", param.myRank), HCCL_E_INTERNAL);
210+ const size_t localRankIdx = static_cast<size_t>(localRankIt - localRanks.begin());
211+ pairRank = remoteRanks[localRankIdx % remoteRanks.size()];
212+ 
213+ const ChannelGroup &remoteGroup = groups[remoteGroupIdx];
214+ const auto getRemoteChannelIdx = [&remoteGroup](uint32_t remoteRank) {
215+ const auto channelIt = std::find_if(
216+ remoteGroup.plans.begin(), remoteGroup.plans.end(), [remoteRank](const ChannelPlan &plan) {
217+ return plan.remoteRank == remoteRank;
218+ });
219+ return static_cast<size_t>(channelIt - remoteGroup.plans.begin());
220+ };
221+ const size_t pairChannel = getRemoteChannelIdx(pairRank);
222+ CHK_PRT_RET(pairChannel >= remoteGroup.plans.size(),
223+ HCCL_ERROR("[AllGather] no layer-1 channel from rank %u to pair rank %u", param.myRank, pairRank),
224+ HCCL_E_INTERNAL);
225+ pairChannelIdx = static_cast<uint32_t>(pairChannel);
226+ 
227+ for (size_t remoteRankIdx = 0; remoteRankIdx < remoteRanks.size(); ++remoteRankIdx) {
228+ if (remoteRankIdx % localRanks.size() == localRankIdx) {
229+ ownedRemoteRanks.push_back(remoteRanks[remoteRankIdx]);
230+ ownedRemoteChannelIdx.push_back(static_cast<uint32_t>(getRemoteChannelIdx(remoteRanks[remoteRankIdx])));
231+ }
232+ }
233+ 
234+ for (size_t targetRankIdx = localRankIdx; targetRankIdx < remoteRanks.size();
235+ targetRankIdx += localRanks.size()) {
236+ const size_t targetChannel = getRemoteChannelIdx(remoteRanks[targetRankIdx]);
237+ CHK_PRT_RET(targetChannel >= remoteGroup.plans.size(),
238+ HCCL_ERROR(
239+ "[AllGather] no stage-2 channel from rank %u to rank %u", param.myRank, remoteRanks[targetRankIdx]),
240+ HCCL_E_INTERNAL);
241+ stageTwoTargetChannelIdx.push_back(static_cast<uint32_t>(targetChannel));
242+ }
243+ HCCL_INFO("[AllGather] rank %u enables hierarchical path: local %zu, remote %zu, stage-1 pair %u, "
244+ "owned remote %zu, stage-2 targets %zu",
245+ param.myRank, localRanks.size(), remoteRanks.size(), pairRank, ownedRemoteRanks.size(),
246+ stageTwoTargetChannelIdx.size());
247+ }
248+ 
249+ // 4*1 递归倍增配对:round-1 = myRank^1,round-2 = myRank^2(rank id 在通信域内为 0..3)
250+ uint32_t pairChannelIdxR2 = 0;
251+ const bool recursiveEnabled = param.rankSize == 4 && remoteGroupIdx < groups.size() && remoteRanks.size() == 3;
252+ if (recursiveEnabled) {
253+ const ChannelGroup &remoteGroup = groups[remoteGroupIdx];
254+ const auto getRemoteChannelIdx = [&remoteGroup](uint32_t remoteRank) {
255+ const auto channelIt = std::find_if(
256+ remoteGroup.plans.begin(), remoteGroup.plans.end(), [remoteRank](const ChannelPlan &plan) {
257+ return plan.remoteRank == remoteRank;
258+ });
259+ return static_cast<size_t>(channelIt - remoteGroup.plans.begin());
260+ };
261+ pairRank = param.myRank ^ 1U;
262+ const size_t pairChannel = getRemoteChannelIdx(pairRank);
263+ CHK_PRT_RET(pairChannel >= remoteGroup.plans.size(),
264+ HCCL_ERROR("[AllGather] no layer-1 channel from rank %u to round-1 pair %u", param.myRank, pairRank),
265+ HCCL_E_INTERNAL);
266+ pairChannelIdx = static_cast<uint32_t>(pairChannel);
267+ const uint32_t pairRankR2 = param.myRank ^ 2U;
268+ const size_t pairChannelR2 = getRemoteChannelIdx(pairRankR2);
269+ CHK_PRT_RET(pairChannelR2 >= remoteGroup.plans.size(),
270+ HCCL_ERROR("[AllGather] no layer-1 channel from rank %u to round-2 pair %u", param.myRank, pairRankR2),
271+ HCCL_E_INTERNAL);
272+ pairChannelIdxR2 = static_cast<uint32_t>(pairChannelR2);
273+ HCCL_INFO("[AllGather] rank %u enables recursive path: round-1 pair %u, round-2 pair %u",
274+ param.myRank, pairRank, pairRankR2);
275+ }
276+ 
277+ std::vector<CcuKernelInfo> kernelInfos(groups.size());
278+ std::vector<CcuKernelInfo> flatKernelInfos(groups.size());
279+ for (size_t idx = 0; idx < groups.size(); ++idx) {
280+ CcuKernelInfo &kernelInfo = kernelInfos[idx];
281+ const int nameRet = std::snprintf(
282+ kernelInfo.kernelFuncName, sizeof(kernelInfo.kernelFuncName), "CcuAllGatherDirectPush_%zu", idx);
283+ CHK_PRT_RET(nameRet <= 0 || static_cast<size_t>(nameRet) >= sizeof(kernelInfo.kernelFuncName),
284+ HCCL_ERROR("[AllGather] Failed to create CCU kernel name"), HCCL_E_INTERNAL);
285+ kernelInfo.kernelFunc = reinterpret_cast<void *>(ops_hccl::CcuAllGatherDirectPush);
286+ 
287+ auto kernelArg = std::make_shared<CcuKernelArgAllGather>();
288+ kernelArg->channelCount = static_cast<uint32_t>(groups[idx].handles.size());
289+ kernelArg->copySelf = (idx == copyKernelIdx) ? 1U : 0U;
290+ kernelArg->isLocalLayer = (idx == localGroupIdx) ? 1U : 0U;
291+ kernelArg->hierarchicalEnabled = hierarchicalEnabled ? 1U : 0U;
292+ kernelArg->myRank = param.myRank;
293+ kernelArg->pairRank = pairRank;
294+ kernelArg->pairChannelIdx = pairChannelIdx;
295+ kernelArg->pairChannelIdxR2 = pairChannelIdxR2;
296+ kernelArg->localRankCount = static_cast<uint32_t>(localRanks.size());
297+ kernelArg->remoteRankCount = static_cast<uint32_t>(remoteRanks.size());
298+ for (size_t rankIdx = 0; rankIdx < localRanks.size(); ++rankIdx) {
299+ kernelArg->localRanks[rankIdx] = localRanks[rankIdx];
300+ }
301+ kernelArg->ownedRemoteCount = static_cast<uint32_t>(ownedRemoteRanks.size());
302+ for (size_t rankIdx = 0; rankIdx < ownedRemoteRanks.size(); ++rankIdx) {
303+ kernelArg->ownedRemoteRanks[rankIdx] = ownedRemoteRanks[rankIdx];
304+ kernelArg->ownedRemoteChannelIdx[rankIdx] = ownedRemoteChannelIdx[rankIdx];
305+ }
306+ kernelArg->stageTwoTargetCount = static_cast<uint32_t>(stageTwoTargetChannelIdx.size());
307+ for (size_t targetIdx = 0; targetIdx < stageTwoTargetChannelIdx.size(); ++targetIdx) {
308+ kernelArg->stageTwoTargetChannelIdx[targetIdx] = stageTwoTargetChannelIdx[targetIdx];
309+ }
310+ for (size_t channelIdx = 0; channelIdx < groups[idx].handles.size(); ++channelIdx) {
311+ kernelArg->channels[channelIdx] = groups[idx].handles[channelIdx];
312+ }
313+ kernelInfo.setKernelArg(kernelArg);
314+ 
315+ // 同一份 kernelArg 再注册一个极简扁平 kernel(小消息/4*1 走它,指令数最小化)
316+ CcuKernelInfo &flatKernelInfo = flatKernelInfos[idx];
317+ const int flatNameRet = std::snprintf(
318+ flatKernelInfo.kernelFuncName, sizeof(flatKernelInfo.kernelFuncName), "CcuAllGatherFlat_%zu", idx);
319+ CHK_PRT_RET(flatNameRet <= 0 || static_cast<size_t>(flatNameRet) >= sizeof(flatKernelInfo.kernelFuncName),
320+ HCCL_ERROR("[AllGather] Failed to create CCU kernel name"), HCCL_E_INTERNAL);
321+ flatKernelInfo.kernelFunc = reinterpret_cast<void *>(ops_hccl::CcuAllGatherFlat);
322+ flatKernelInfo.setKernelArg(kernelArg);
323+ }
324+ 
325+ CcuResult ccuRet = HcommCcuKernelRegisterStart(insHandle);
326+ if (ccuRet != CCU_SUCCESS) {
327+ HCCL_ERROR("[AllGather] CCU kernel register start failed: %d", ccuRet);
328+ return ConvertCcuResult(ccuRet);
329+ }
330+ 
331+ // 每条 channel 只能属于一个 kernel:分层拓扑只注册分层 kernel(内含扁平整写分支),
332+ // 非分层拓扑只注册极简扁平 kernel(含 4*1 递归两轮)
333+ const bool registerHier = hierarchicalEnabled;
334+ if (registerHier) {
335+ resCtx.ccuKernels.resize(kernelInfos.size());
336+ } else {
337+ resCtx.flatKernels.resize(flatKernelInfos.size());
338+ }
339+ for (size_t idx = 0; idx < kernelInfos.size(); ++idx) {
340+ if (registerHier) {
341+ const void *kernelArgs[] = {kernelInfos[idx].kernelArg};
342+ ccuRet = HcommCcuKernelRegister(insHandle, RESERVED_DIE_ID, kernelInfos[idx].kernelFuncName,
343+ kernelInfos[idx].kernelFunc, kernelArgs, KERNEL_ARG_NUM, &resCtx.ccuKernels[idx]);
344+ if (ccuRet != CCU_SUCCESS) {
345+ HCCL_ERROR("[AllGather] CCU kernel %zu register failed: %d", idx, ccuRet);
346+ return ConvertCcuResult(ccuRet);
347+ }
348+ } else {
349+ const void *flatKernelArgs[] = {flatKernelInfos[idx].kernelArg};
350+ ccuRet = HcommCcuKernelRegister(insHandle, RESERVED_DIE_ID, flatKernelInfos[idx].kernelFuncName,
351+ flatKernelInfos[idx].kernelFunc, flatKernelArgs, KERNEL_ARG_NUM, &resCtx.flatKernels[idx]);
352+ if (ccuRet != CCU_SUCCESS) {
353+ HCCL_ERROR("[AllGather] CCU flat kernel %zu register failed: %d", idx, ccuRet);
354+ return ConvertCcuResult(ccuRet);
355+ }
356+ }
357+ }
358+ 
359+ ccuRet = HcommCcuKernelRegisterEnd(insHandle);
360+ if (ccuRet != CCU_SUCCESS) {
361+ HCCL_ERROR("[AllGather] CCU kernel register end failed: %d", ccuRet);
362+ return ConvertCcuResult(ccuRet);
363+ }
364+ return HCCL_SUCCESS;
365+}
366+ 
367+HcclResult CreateResources(HcclComm comm, const OpParam &param, AlgResourceCtx &resCtx)
368+{
369+ resCtx.ccuThread = param.cpuThread;
370+ void *cclBufferAddr = nullptr;
371+ uint64_t cclBufferSize = 0;
372+ CHK_RET(HcclGetHcclBuffer(comm, &cclBufferAddr, &cclBufferSize));
373+ resCtx.localBuffer = CommBuffer{cclBufferAddr, cclBufferSize};
374+ resCtx.threads.push_back(param.cpuThread);
375+ 
376+ if (param.rankSize == 1) {
377+ return HCCL_SUCCESS;
378+ }
379+ 
380+ std::vector<ChannelGroup> groups;
381+ CHK_RET(BuildChannelGroups(comm, param, groups));
382+ if (groups.size() == MAX_KERNEL_NUM) {
383+ resCtx.threads.resize(MAX_KERNEL_NUM);
384+ CHK_RET(HcclThreadAcquire(comm, CommEngine::COMM_ENGINE_CCU, 1, 1, &resCtx.threads[1]));
385+ }
386+ CHK_RET(RegisterKernels(comm, param, groups, resCtx));
387+ return HCCL_SUCCESS;
388+}
389+ 
390+// 每个通信域的静态信息缓存:comm 创建后 rank/dfx 信息不变,避免每 call 重复 HCCL API 查询
391+struct CommStaticInfo {
392+ uint32_t myRank = INVALID_VALUE_RANKID;
393+ uint32_t rankSize = 0;
394+ aclrtStream stream = nullptr;
395+ ThreadHandle cpuThread = 0;
396+};
397+ 
398+std::mutex g_commInfoMtx;
399+std::map<HcclComm, CommStaticInfo> g_commInfoCache;
400+ 
401+HcclResult GetCommStaticInfo(HcclComm comm, aclrtStream stream, CommEngine engine, CommStaticInfo &info)
402+{
403+ {
404+ std::lock_guard<std::mutex> lock(g_commInfoMtx);
405+ const auto it = g_commInfoCache.find(comm);
406+ if (it != g_commInfoCache.end() && it->second.stream == stream) {
407+ info = it->second;
408+ return HCCL_SUCCESS;
409+ }
410+ }
411+ // 注册算子信息(每通信域一次)
412+ HcclDfxOpInfo dfxInfo{};
413+ char commName[COMM_INDENTIFIER_MAX_LENGTH];
414+ CHK_RET(HcclGetCommName(comm, commName));
415+ CHK_RET(HcclDfxRegOpInfoByCommId(commName, reinterpret_cast<void *>(&dfxInfo)));
416+ 
417+ CommStaticInfo fresh;
418+ CHK_RET(HcclGetRankId(comm, &fresh.myRank));
419+ CHK_RET(HcclGetRankSize(comm, &fresh.rankSize));
420+ fresh.stream = stream;
421+ // 将用户传入的 stream 转换为 CCU 通信引擎中的 thread,并申请 1 个 notify
422+ CHK_RET(HcclThreadAcquireWithStream(comm, engine, stream, 1, &fresh.cpuThread));
423+ 
424+ std::lock_guard<std::mutex> lock(g_commInfoMtx);
425+ g_commInfoCache[comm] = fresh;
426+ info = fresh;
427+ return HCCL_SUCCESS;
428+}
429+ 
430+} // namespace
431+ 
432+HcclResult HcclAllGather(
433+ void *sendBuf, void *recvBuf, uint64_t sendCount, HcclDataType dataType, HcclComm comm, aclrtStream stream)
434+{
435+ CHK_PTR_NULL(sendBuf);
436+ CHK_PTR_NULL(recvBuf);
437+ CHK_PTR_NULL(comm);
438+ CHK_PTR_NULL(stream);
439+ 
440+ // 构造算子参数
441+ OpParam param;
442+ const int tagRet = std::snprintf(param.tag, sizeof(param.tag), "%s", "hccl_custom_allgather");
443+ CHK_PRT_RET(tagRet <= 0 || static_cast<size_t>(tagRet) >= sizeof(param.tag),
444+ HCCL_ERROR("[AllGather] Failed to fill operation tag"), HCCL_E_INTERNAL);
445+ param.inputPtr = sendBuf;
446+ param.outputPtr = recvBuf;
447+ param.count = sendCount;
448+ param.dataType = dataType;
449+ param.opType = HcclCMDType::HCCL_CMD_ALLGATHER;
450+ 
451+ // ==============================================
452+ // STEP 1: 解析拓扑信息(每通信域缓存)
453+ // ==============================================
454+ CommEngine ccuEngine = CommEngine::COMM_ENGINE_CCU;
455+ CommStaticInfo commInfo;
456+ CHK_RET(GetCommStaticInfo(comm, stream, ccuEngine, commInfo));
457+ param.myRank = commInfo.myRank;
458+ param.rankSize = commInfo.rankSize;
459+ param.cpuThread = commInfo.cpuThread;
460+ CHK_PRT_RET(param.rankSize == 0 || param.rankSize > MAX_RANK_SIZE || param.myRank >= param.rankSize,
461+ HCCL_ERROR("[AllGather] Invalid rank: myRank %u, rankSize %u", param.myRank, param.rankSize), HCCL_E_PARA);
462+ CHK_PRT_RET(SIZE_TABLE.find(dataType) == SIZE_TABLE.end(),
463+ HCCL_ERROR("[AllGather] Unsupported data type %d", dataType), HCCL_E_NOT_SUPPORT);
464+ 
465+ void *ctx = nullptr;
466+ uint64_t size = 0;
467+ if (HcclEngineCtxGet(comm, param.tag, ccuEngine, &ctx, &size) == HCCL_SUCCESS) {
468+ // CCU 资源已经存在,复用资源
469+ HCCL_INFO("Engine context already exists");
470+ param.resCtx = ctx;
471+ param.ctxSize = size;
472+ } else {
473+ // Device 资源不存在,资源构建
474+ AlgResourceCtx resCtxHost{};
475+ CHK_RET(CreateResources(comm, param, resCtxHost));
476+ 
477+ // ==============================================
478+ // STEP 2.3: 申请通信引擎上下文
479+ // ==============================================
480+ // 申请 CCU 通信引擎上下文,存放 AlgResourceCtx 信息
481+ std::vector<char> seq = resCtxHost.Serialize();
482+ uint64_t seqSize = seq.size();
483+ param.ctxSize = seqSize;
484+ CHK_RET(HcclEngineCtxCreate(comm, param.tag, ccuEngine, param.ctxSize, &param.resCtx));
485+ CHK_RET(HcclEngineCtxCopy(comm, ccuEngine, param.tag, seq.data(), seqSize, 0));
486+ }
487+ 
488+ // ==============================================
489+ // STEP 3: 下发 CCU Kernel
490+ // ==============================================
491+ CHK_RET(ops_hccl::ExecOp(param));
492+ return HCCL_SUCCESS;
493+}
@@ -0,0 +1,335 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#include <algorithm>
12+#include <limits>
13+#include <map>
14+#include <mutex>
15+#include <vector>
16+ 
17+#include <ccu/ccu_launch.h>
18+#include <ccu/ccu_res.h>
19+ 
20+#include "common.h"
21+#include "custom.h"
22+#include "exec_op.h"
23+#include "log.h"
24+ 
25+namespace ops_hccl {
26+ 
27+namespace {
28+ 
29+ constexpr uint32_t THREAD_NOTIFY_IDX = 0;
30+ constexpr uint32_t TASK_ARG_NUM = 13;
31+ constexpr uint32_t FLAT_TASK_ARG_NUM = 10;
32+ constexpr uint64_t HIERARCHICAL_PATH_THRESHOLD = 1024 * 1024;
33+ constexpr uint64_t HIERARCHICAL_SPLIT_ALIGN = 4096;
34+ constexpr uint64_t RECURSIVE_PATH_THRESHOLD = 1024 * 1024;
35+ 
36+ std::mutex g_callCountMtx;
37+ std::map<const void *, uint64_t> g_callCount;
38+ 
39+ HcclResult ConvertCcuResult(CcuResult result)
40+ {
41+ switch (result) {
42+ case CCU_SUCCESS:
43+ return HCCL_SUCCESS;
44+ case CCU_E_PARA:
45+ return HCCL_E_PARA;
46+ case CCU_E_PTR:
47+ return HCCL_E_PTR;
48+ case CCU_E_INTERNAL:
49+ return HCCL_E_INTERNAL;
50+ case CCU_E_NOT_SUPPORT:
51+ return HCCL_E_NOT_SUPPORT;
52+ case CCU_E_NOT_FOUND:
53+ return HCCL_E_NOT_FOUND;
54+ case CCU_E_UNAVAIL:
55+ return HCCL_E_UNAVAIL;
56+ default:
57+ return HCCL_E_INTERNAL;
58+ }
59+ }
60+ 
61+ HcclResult LaunchKernel(
62+ ThreadHandle thread, CcuKernelHandle kernel, const std::vector<uint64_t> &taskArgs, uint32_t argNum)
63+ {
64+ CcuResult result = HcommCcuKernelLaunch(thread, kernel, taskArgs.data(), argNum);
65+ if (result != CCU_SUCCESS) {
66+ HCCL_ERROR("[AllGather] CCU kernel launch failed: %d", result);
67+ return ConvertCcuResult(result);
68+ }
69+ return HCCL_SUCCESS;
70+ }
71+ 
72+ // 每次 launch 取一个全局递增序号:taskArgs[11]=是否首轮(显式交换地址/token),
73+ // taskArgs[12]=序号奇偶(PostSync 的 notify 位随之轮换,防多打一坍缩)
74+ uint64_t NextLaunchSeq(const OpParam &param)
75+ {
76+ std::lock_guard<std::mutex> lock(g_callCountMtx);
77+ return g_callCount[param.resCtx]++;
78+ }
79+ 
80+ HcclResult PreSyncThreads(ThreadHandle mainThread, ThreadHandle slaveThread)
81+ {
82+ CHK_RET(static_cast<HcclResult>(HcommThreadNotifyRecordOnThread(mainThread, slaveThread, THREAD_NOTIFY_IDX)));
83+ CHK_RET(static_cast<HcclResult>(HcommThreadNotifyWaitOnThread(slaveThread, THREAD_NOTIFY_IDX, CUSTOM_TIMEOUT)));
84+ return HCCL_SUCCESS;
85+ }
86+ 
87+ HcclResult PostSyncThreads(ThreadHandle mainThread, ThreadHandle slaveThread)
88+ {
89+ CHK_RET(static_cast<HcclResult>(HcommThreadNotifyWaitOnThread(mainThread, THREAD_NOTIFY_IDX, CUSTOM_TIMEOUT)));
90+ CHK_RET(static_cast<HcclResult>(HcommThreadNotifyRecordOnThread(slaveThread, mainThread, THREAD_NOTIFY_IDX)));
91+ return HCCL_SUCCESS;
92+ }
93+ 
94+ HcclResult LaunchAllKernels(const OpParam &param, const AlgResourceCtx &resCtx,
95+ const std::vector<uint64_t> &taskArgs, uint32_t argNum, const std::vector<CcuKernelHandle> &kernels)
96+ {
97+ if (kernels.size() == 1) {
98+ return LaunchKernel(param.cpuThread, kernels[0], taskArgs, argNum);
99+ }
100+ 
101+ const ThreadHandle slaveThread = resCtx.threads[1];
102+ CHK_RET(PreSyncThreads(param.cpuThread, slaveThread));
103+ CHK_RET(LaunchKernel(param.cpuThread, kernels[0], taskArgs, argNum));
104+ CHK_RET(LaunchKernel(slaveThread, kernels[1], taskArgs, argNum));
105+ CHK_RET(PostSyncThreads(param.cpuThread, slaveThread));
106+ return HCCL_SUCCESS;
107+ }
108+ 
109+ // 反序列化结果缓存:引擎上下文一次构建后不变,避免每 call 重复拷贝解析
110+ std::mutex g_resCtxMtx;
111+ std::map<const void *, AlgResourceCtx> g_resCtxCache;
112+ 
113+ const AlgResourceCtx &GetResCtx(const OpParam &param)
114+ {
115+ std::lock_guard<std::mutex> lock(g_resCtxMtx);
116+ const auto it = g_resCtxCache.find(param.resCtx);
117+ if (it != g_resCtxCache.end()) {
118+ return it->second;
119+ }
120+ char *ctx = static_cast<char *>(param.resCtx);
121+ std::vector<char> seq(ctx, ctx + param.ctxSize);
122+ AlgResourceCtx resCtx;
123+ resCtx.DeSerialize(seq);
124+ const auto result = g_resCtxCache.emplace(param.resCtx, std::move(resCtx));
125+ return result.first->second;
126+ }
127+ 
128+ // 内存 token 缓存:harness buffer 同址复用,避免每 call 两次驱动查询
129+ std::mutex g_tokenMtx;
130+ std::map<std::pair<uint64_t, uint64_t>, uint64_t> g_tokenCache;
131+ 
132+ CcuResult GetMemTokenCached(uint64_t va, uint64_t size, uint64_t &token)
133+ {
134+ const auto key = std::make_pair(va, size);
135+ {
136+ std::lock_guard<std::mutex> lock(g_tokenMtx);
137+ const auto it = g_tokenCache.find(key);
138+ if (it != g_tokenCache.end()) {
139+ token = it->second;
140+ return CCU_SUCCESS;
141+ }
142+ }
143+ uint64_t tokenInfo = 0;
144+ const CcuResult ccuRet = HcommCcuGetMemToken(va, size, &tokenInfo);
145+ if (ccuRet != CCU_SUCCESS) {
146+ return ccuRet;
147+ }
148+ std::lock_guard<std::mutex> lock(g_tokenMtx);
149+ g_tokenCache[key] = tokenInfo;
150+ token = tokenInfo;
151+ return CCU_SUCCESS;
152+ }
153+ 
154+ // 地址/token 交换判定:全通信域首次或输入/输出 VA 变化时,下一次 launch 先显式交换
155+ std::mutex g_vaMtx;
156+ std::map<const void *, std::pair<uint64_t, uint64_t>> g_vaCache;
157+ 
158+ bool ConsumeExchangeFlag(const OpParam &param, uint64_t inputAddr, uint64_t outputAddr)
159+ {
160+ std::lock_guard<std::mutex> lock(g_vaMtx);
161+ const auto va = std::make_pair(inputAddr, outputAddr);
162+ const auto it = g_vaCache.find(param.resCtx);
163+ if (it == g_vaCache.end() || it->second != va) {
164+ g_vaCache[param.resCtx] = va;
165+ return true;
166+ }
167+ return false;
168+ }
169+ 
170+} // namespace
171+ 
172+HcclResult ExecOp(const OpParam &param)
173+{
174+ const AlgResourceCtx &resCtx = GetResCtx(param);
175+ 
176+ const auto sizeIter = SIZE_TABLE.find(param.dataType);
177+ CHK_PRT_RET(sizeIter == SIZE_TABLE.end(), HCCL_ERROR("[AllGather] Unsupported data type %d", param.dataType),
178+ HCCL_E_NOT_SUPPORT);
179+ CHK_PRT_RET(param.count > std::numeric_limits<uint64_t>::max() / sizeIter->second,
180+ HCCL_ERROR("[AllGather] Input size overflow"), HCCL_E_PARA);
181+ const uint64_t dataSize = param.count * sizeIter->second;
182+ if (dataSize == 0) {
183+ return HCCL_SUCCESS;
184+ }
185+ CHK_PRT_RET(param.rankSize == 0 || dataSize > std::numeric_limits<uint64_t>::max() / param.rankSize,
186+ HCCL_ERROR("[AllGather] Output size overflow"), HCCL_E_PARA);
187+ const uint64_t outputSize = dataSize * param.rankSize;
188+ 
189+ CHK_PRT_RET(resCtx.threads.empty(), HCCL_ERROR("[AllGather] No CCU thread in resource context"), HCCL_E_INTERNAL);
190+ if (param.rankSize == 1) {
191+ return static_cast<HcclResult>(
192+ HcommLocalCopyOnThread(param.cpuThread, param.outputPtr, param.inputPtr, dataSize));
193+ }
194+ // 分层拓扑用 ccuKernels,非分层拓扑用 flatKernels,二者必其一与 threads 对应
195+ const bool useFlatKernels = !resCtx.flatKernels.empty();
196+ const std::vector<CcuKernelHandle> &activeKernels = useFlatKernels ? resCtx.flatKernels : resCtx.ccuKernels;
197+ CHK_PRT_RET(activeKernels.empty() || activeKernels.size() != resCtx.threads.size() || activeKernels.size() > 2,
198+ HCCL_ERROR("[AllGather] Invalid resources: %zu hier kernels, %zu flat kernels, %zu threads",
199+ resCtx.ccuKernels.size(), resCtx.flatKernels.size(), resCtx.threads.size()),
200+ HCCL_E_INTERNAL);
201+ 
202+ const uint64_t inputAddr = reinterpret_cast<uint64_t>(param.inputPtr);
203+ const uint64_t outputAddr = reinterpret_cast<uint64_t>(param.outputPtr);
204+ uint64_t inputToken = 0;
205+ uint64_t outputToken = 0;
206+ CcuResult ccuRet = GetMemTokenCached(inputAddr, dataSize, inputToken);
207+ if (ccuRet != CCU_SUCCESS) {
208+ HCCL_ERROR("[AllGather] Failed to get input token: %d", ccuRet);
209+ return ConvertCcuResult(ccuRet);
210+ }
211+ ccuRet = GetMemTokenCached(outputAddr, outputSize, outputToken);
212+ if (ccuRet != CCU_SUCCESS) {
213+ HCCL_ERROR("[AllGather] Failed to get output token: %d", ccuRet);
214+ return ConvertCcuResult(ccuRet);
215+ }
216+ 
217+ uint64_t processedSize = 0;
218+ bool exchangePending = ConsumeExchangeFlag(param, inputAddr, outputAddr);
219+ while (processedSize < dataSize) {
220+ const uint64_t sliceSize = std::min<uint64_t>(MAX_DATA_SIZE, dataSize - processedSize);
221+ const uint64_t outputOffset = dataSize * param.myRank;
222+ const uint64_t localBlockOffset = (static_cast<uint64_t>(param.myRank) / 8) * 8 * dataSize;
223+ bool useHierarchicalPath = (param.rankSize == 16 || param.rankSize == 12) && resCtx.ccuKernels.size() == 2
224+ && sliceSize >= HIERARCHICAL_PATH_THRESHOLD;
225+ uint64_t collectPartSize = 0;
226+ uint64_t directPartSize = 0;
227+ if (useHierarchicalPath) {
228+ // 三路拆分(LP 最优):2*8 = A4:D3:S2 4(15/11 S/B);8+4 = A3:D2:S2 2(11/7 S/B)
229+ const uint64_t collectNumer = (param.rankSize == 16) ? 4 : 3;
230+ const uint64_t directNumer = (param.rankSize == 16) ? 3 : 2;
231+ const uint64_t denom = (param.rankSize == 16) ? 11 : 7;
232+ collectPartSize = ((sliceSize * collectNumer / denom) / HIERARCHICAL_SPLIT_ALIGN)
233+ * HIERARCHICAL_SPLIT_ALIGN;
234+ directPartSize = ((sliceSize * directNumer / denom) / HIERARCHICAL_SPLIT_ALIGN)
235+ * HIERARCHICAL_SPLIT_ALIGN;
236+ if (collectPartSize == 0 || directPartSize == 0
237+ || collectPartSize + directPartSize >= sliceSize) {
238+ collectPartSize = 0;
239+ directPartSize = 0;
240+ useHierarchicalPath = false;
241+ }
242+ }
243+ const uint64_t relayPartSize = sliceSize - collectPartSize - directPartSize;
244+ 
245+ const bool useRecursivePath = !useHierarchicalPath && param.rankSize == 4
246+ && resCtx.flatKernels.size() == 1 && sliceSize >= RECURSIVE_PATH_THRESHOLD;
247+ if (useRecursivePath) {
248+ // 4*1 大消息递归倍增(单 launch 融合两轮,内部双 barrier 分用 even/odd 位):
249+ // 占两个 launch 序号保持全局奇偶纪律
250+ const uint64_t seq = NextLaunchSeq(param);
251+ (void)NextLaunchSeq(param);
252+ std::vector<uint64_t> taskArgs = {
253+ inputAddr + processedSize,
254+ outputAddr + processedSize,
255+ inputToken,
256+ outputToken,
257+ sliceSize,
258+ outputOffset,
259+ TRANSFER_ROUND_FUSED,
260+ (static_cast<uint64_t>(param.myRank) & ~1ULL) * dataSize, // pairBaseOffset
261+ exchangePending ? 1ULL : 0ULL,
262+ seq & 1,
263+ };
264+ exchangePending = false;
265+ CHK_RET(LaunchAllKernels(param, resCtx, taskArgs, FLAT_TASK_ARG_NUM, resCtx.flatKernels));
266+ } else if (!useHierarchicalPath && useFlatKernels) {
267+ // 极简扁平 kernel(10 个 taskArgs),arg7 保留为 0
268+ const uint64_t seq = NextLaunchSeq(param);
269+ std::vector<uint64_t> taskArgs = {
270+ inputAddr + processedSize,
271+ outputAddr + processedSize,
272+ inputToken,
273+ outputToken,
274+ sliceSize,
275+ outputOffset,
276+ TRANSFER_FLAT,
277+ 0,
278+ exchangePending ? 1ULL : 0ULL,
279+ seq & 1,
280+ };
281+ exchangePending = false;
282+ CHK_RET(LaunchAllKernels(param, resCtx, taskArgs, FLAT_TASK_ARG_NUM, resCtx.flatKernels));
283+ } else if (!useHierarchicalPath) {
284+ // 分层 kernel 的扁平整写分支(13 个 taskArgs,mode=TRANSFER_FLAT)
285+ const uint64_t seq = NextLaunchSeq(param);
286+ std::vector<uint64_t> taskArgs = {
287+ inputAddr + processedSize,
288+ outputAddr + processedSize,
289+ inputToken,
290+ outputToken,
291+ sliceSize,
292+ outputOffset,
293+ TRANSFER_FLAT,
294+ 0,
295+ 0,
296+ 0,
297+ localBlockOffset,
298+ exchangePending ? 1ULL : 0ULL,
299+ seq & 1,
300+ };
301+ exchangePending = false;
302+ CHK_RET(LaunchAllKernels(param, resCtx, taskArgs, TASK_ARG_NUM, resCtx.ccuKernels));
303+ } else {
304+ std::vector<uint64_t> taskArgs = {
305+ inputAddr + processedSize,
306+ outputAddr + processedSize,
307+ inputToken,
308+ outputToken,
309+ sliceSize,
310+ outputOffset,
311+ TRANSFER_FLAT,
312+ collectPartSize,
313+ directPartSize,
314+ relayPartSize,
315+ localBlockOffset,
316+ 0, // exchangeFlag:仅本轮首个 launch 按需置 1
317+ 0, // syncParity:notify 位按 launch 奇偶轮换
318+ };
319+ const uint64_t seqOne = NextLaunchSeq(param);
320+ taskArgs[6] = TRANSFER_PHASE_ONE;
321+ taskArgs[11] = exchangePending ? 1 : 0;
322+ taskArgs[12] = seqOne & 1;
323+ exchangePending = false;
324+ CHK_RET(LaunchAllKernels(param, resCtx, taskArgs, TASK_ARG_NUM, resCtx.ccuKernels));
325+ const uint64_t seqTwo = NextLaunchSeq(param);
326+ taskArgs[6] = TRANSFER_PHASE_TWO;
327+ taskArgs[12] = seqTwo & 1;
328+ CHK_RET(LaunchAllKernels(param, resCtx, taskArgs, TASK_ARG_NUM, resCtx.ccuKernels));
329+ }
330+ processedSize += sliceSize;
331+ }
332+ 
333+ return HCCL_SUCCESS;
334+}
335+} // namespace ops_hccl
@@ -0,0 +1,21 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_CCU_EXEC_OP_H
12+#define OPS_HCCL_CCU_EXEC_OP_H
13+ 
14+#include <hccl/hcomm_primitives.h>
15+#include "common.h"
16+ 
17+namespace ops_hccl {
18+// 执行算法任务编排
19+HcclResult ExecOp(const OpParam &param);
20+} // namespace ops_hccl
21+#endif // OPS_HCCL_CCU_EXEC_OP_H
@@ -0,0 +1,18 @@
1+ # -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+if(TARGET hccl)
12+ target_sources(hccl PRIVATE
13+ ${CMAKE_CURRENT_SOURCE_DIR}/ccu_kernel.cc
14+ )
15+ target_include_directories(hccl PRIVATE
16+ ${CMAKE_CURRENT_SOURCE_DIR}
17+ )
18+endif()
@@ -0,0 +1,568 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#include <cstdint>
12+#include <vector>
13+ 
14+#include "common.h"
15+#include "custom.h"
16+#include "ccu_kernel.h"
17+#include "log.h"
18+ 
19+#define RETURN_IF_CCU_ERROR(call) \
20+ do { \
21+ CcuResult ccuCallResult = (call); \
22+ if (ccuCallResult != CCU_SUCCESS) { \
23+ return ccuCallResult; \
24+ } \
25+ } while (0)
26+ 
27+namespace ops_hccl {
28+ 
29+namespace ccu = ::AscendC::ccu;
30+ 
31+namespace {
32+ 
33+ constexpr uint32_t REMOTE_OUTPUT_VAR_ID = 1;
34+ constexpr uint32_t REMOTE_TOKEN_VAR_ID = 2;
35+ constexpr uint32_t CHANNEL_NOTIFY_IDX = 0;
36+ // notify 位分配(slot 0):地址/token 交换用 bit1|bit2(仅首轮或 VA 变化时);
37+ // 每次 launch 的 PostSync 为普通 NotifyRecord,在 bit3 与 bit4 之间按 launch 奇偶轮换,
38+ // 保证同一 bit 的 record/wait 中间至少隔一次完整 launch(防多打一坍缩)。
39+ constexpr uint32_t EXCHANGE_MASK = (1U << REMOTE_OUTPUT_VAR_ID) | (1U << REMOTE_TOKEN_VAR_ID);
40+ constexpr uint16_t POST_MASK_EVEN = 1U << 3;
41+ constexpr uint16_t POST_MASK_ODD = 1U << 4;
42+ 
43+ struct DirectPushContext {
44+ const CcuKernelArgAllGather *arg = nullptr;
45+ ccu::Variable input;
46+ ccu::Variable output;
47+ ccu::Variable inputToken;
48+ ccu::Variable outputToken;
49+ ccu::Variable dataSize;
50+ ccu::Variable outputOffset;
51+ ccu::Variable mode;
52+ ccu::Variable collectPartSize; // A:服内收集 + phase-2 转发
53+ ccu::Variable directPartSize; // D:phase-1 扁平直推
54+ ccu::Variable relayPartSize; // S2:写 pair + 服内广播
55+ ccu::Variable localBlockOffset;
56+ ccu::Variable firstCall;
57+ ccu::Variable syncParity;
58+ std::vector<ccu::Variable> remoteOutput;
59+ std::vector<ccu::Variable> remoteToken;
60+ ccu::Event event;
61+ };
62+ 
63+ CcuResult InitResources(DirectPushContext &ctx)
64+ {
65+ ctx.remoteOutput.resize(ctx.arg->channelCount);
66+ ctx.remoteToken.resize(ctx.arg->channelCount);
67+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
68+ ctx.remoteOutput[channelIdx]
69+ = ccu::GetResByChannel<ccu::Variable>(ctx.arg->channels[channelIdx], REMOTE_OUTPUT_VAR_ID);
70+ ctx.remoteToken[channelIdx]
71+ = ccu::GetResByChannel<ccu::Variable>(ctx.arg->channels[channelIdx], REMOTE_TOKEN_VAR_ID);
72+ }
73+ return CCU_SUCCESS;
74+ }
75+ 
76+ CcuResult LoadTaskArgs(DirectPushContext &ctx)
77+ {
78+ uint32_t argIdx = 0;
79+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.input, argIdx++));
80+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.output, argIdx++));
81+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.inputToken, argIdx++));
82+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.outputToken, argIdx++));
83+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.dataSize, argIdx++));
84+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.outputOffset, argIdx++));
85+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.mode, argIdx++));
86+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.collectPartSize, argIdx++));
87+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.directPartSize, argIdx++));
88+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.relayPartSize, argIdx++));
89+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.localBlockOffset, argIdx++));
90+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.firstCall, argIdx++));
91+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.syncParity, argIdx++));
92+ return CCU_SUCCESS;
93+ }
94+ 
95+ // 首轮显式交换本端 recvBuf 地址与 token;后续轮由 PostSync 捎带,无需本步骤
96+ CcuResult ExchangeRemoteOutput(DirectPushContext &ctx)
97+ {
98+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
99+ RETURN_IF_CCU_ERROR(ccu::WriteVariableWithNotify(ctx.arg->channels[channelIdx], ctx.output,
100+ REMOTE_OUTPUT_VAR_ID, CHANNEL_NOTIFY_IDX, 1U << REMOTE_OUTPUT_VAR_ID));
101+ RETURN_IF_CCU_ERROR(ccu::WriteVariableWithNotify(ctx.arg->channels[channelIdx], ctx.outputToken,
102+ REMOTE_TOKEN_VAR_ID, CHANNEL_NOTIFY_IDX, 1U << REMOTE_TOKEN_VAR_ID));
103+ }
104+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
105+ RETURN_IF_CCU_ERROR(
106+ ccu::NotifyWait(ctx.arg->channels[channelIdx], CHANNEL_NOTIFY_IDX, EXCHANGE_MASK));
107+ }
108+ return CCU_SUCCESS;
109+ }
110+ 
111+ // 后同步:普通 NotifyRecord/Wait,等待对端写齐 = 确认对端本阶段数据已全部落盘。
112+ // 地址/token 只在首轮或 VA 变化时经 ExchangeRemoteOutput 刷新(harness buffer 同址复用),
113+ // notify 位由调用方按 launch 奇偶指定,保证同一 bit 的 record/wait 严格交替(防多打一坍缩)
114+ CcuResult PostSync(DirectPushContext &ctx, uint16_t syncMask)
115+ {
116+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
117+ RETURN_IF_CCU_ERROR(
118+ ccu::NotifyRecord(ctx.arg->channels[channelIdx], CHANNEL_NOTIFY_IDX, syncMask));
119+ }
120+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
121+ RETURN_IF_CCU_ERROR(ccu::NotifyWait(ctx.arg->channels[channelIdx], CHANNEL_NOTIFY_IDX, syncMask));
122+ }
123+ return CCU_SUCCESS;
124+ }
125+ 
126+ // 信用后移(小消息):本轮只 record,wait 推迟到下一轮开头——信号飞行时间被整轮覆盖,
127+ // 每轮省一次完整 RT。上一轮 notify 位与本论奇偶相反。
128+ CcuResult PostRecordOnly(DirectPushContext &ctx, uint16_t syncMask)
129+ {
130+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
131+ RETURN_IF_CCU_ERROR(
132+ ccu::NotifyRecord(ctx.arg->channels[channelIdx], CHANNEL_NOTIFY_IDX, syncMask));
133+ }
134+ return CCU_SUCCESS;
135+ }
136+ 
137+ CcuResult PostWaitOnly(DirectPushContext &ctx, uint16_t syncMask)
138+ {
139+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
140+ RETURN_IF_CCU_ERROR(ccu::NotifyWait(ctx.arg->channels[channelIdx], CHANNEL_NOTIFY_IDX, syncMask));
141+ }
142+ return CCU_SUCCESS;
143+ }
144+ 
145+ uint16_t GetCompleteMask(uint32_t operationCount)
146+ {
147+ return (operationCount >= 16U) ? 0xFFFFU : static_cast<uint16_t>((1U << operationCount) - 1U);
148+ }
149+ 
150+ // 扁平直推(整写版):小消息与 4*1 拓扑(单波时延最优)
151+ CcuResult DirectTransferWhole(DirectPushContext &ctx)
152+ {
153+ ccu::LocalAddr src;
154+ src.addr = ctx.input;
155+ src.token = ctx.inputToken;
156+ 
157+ std::vector<ccu::RemoteAddr> remoteDst(ctx.arg->channelCount);
158+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
159+ remoteDst[channelIdx].addr = ctx.remoteOutput[channelIdx];
160+ remoteDst[channelIdx].addr += ctx.outputOffset;
161+ remoteDst[channelIdx].token = ctx.remoteToken[channelIdx];
162+ const uint16_t mask = static_cast<uint16_t>(1U << channelIdx);
163+ RETURN_IF_CCU_ERROR(
164+ ccu::Write(ctx.arg->channels[channelIdx], remoteDst[channelIdx], src, ctx.dataSize, ctx.event, mask));
165+ }
166+ 
167+ uint32_t operationCount = ctx.arg->channelCount;
168+ if (ctx.arg->copySelf != 0) {
169+ ccu::LocalAddr localDst;
170+ localDst.addr = ctx.output;
171+ localDst.addr += ctx.outputOffset;
172+ localDst.token = ctx.outputToken;
173+ const uint16_t localMask = static_cast<uint16_t>(1U << operationCount);
174+ RETURN_IF_CCU_ERROR(ccu::LocalCopy(localDst, src, ctx.dataSize, ctx.event, localMask));
175+ ++operationCount;
176+ }
177+ 
178+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, GetCompleteMask(operationCount)));
179+ return CCU_SUCCESS;
180+ }
181+ 
182+ // 4*1 递归倍增 round-1:与 r^1 交换 own 切片 + own 本地落盘
183+ CcuResult RoundOneTransfer(DirectPushContext &ctx)
184+ {
185+ ccu::LocalAddr src;
186+ src.addr = ctx.input;
187+ src.token = ctx.inputToken;
188+ 
189+ ccu::RemoteAddr dst;
190+ dst.addr = ctx.remoteOutput[ctx.arg->pairChannelIdx];
191+ dst.addr += ctx.outputOffset;
192+ dst.token = ctx.remoteToken[ctx.arg->pairChannelIdx];
193+ RETURN_IF_CCU_ERROR(
194+ ccu::Write(ctx.arg->channels[ctx.arg->pairChannelIdx], dst, src, ctx.dataSize, ctx.event, 1));
195+ 
196+ ccu::LocalAddr localDst;
197+ localDst.addr = ctx.output;
198+ localDst.addr += ctx.outputOffset;
199+ localDst.token = ctx.outputToken;
200+ RETURN_IF_CCU_ERROR(ccu::LocalCopy(localDst, src, ctx.dataSize, ctx.event, 2));
201+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, 3));
202+ return CCU_SUCCESS;
203+ }
204+ 
205+ // 4*1 递归倍增 round-2:把相邻两片(pairBaseOffset 起,各 dataSize)分两条并行写给 r^2,
206+ // 源为本端 recvBuf(own 与 r^1 的切片已于 round-1 落盘)。
207+ // 注:拆两条 S 写以排除单条 2S 大写的速率损失
208+ CcuResult RoundTwoTransfer(DirectPushContext &ctx)
209+ {
210+ ccu::LocalAddr src;
211+ src.addr = ctx.output;
212+ src.addr += ctx.collectPartSize; // pairBaseOffset
213+ src.token = ctx.outputToken;
214+ 
215+ ccu::RemoteAddr dst;
216+ dst.addr = ctx.remoteOutput[ctx.arg->pairChannelIdxR2];
217+ dst.addr += ctx.collectPartSize; // pairBaseOffset
218+ dst.token = ctx.remoteToken[ctx.arg->pairChannelIdxR2];
219+ 
220+ RETURN_IF_CCU_ERROR(
221+ ccu::Write(ctx.arg->channels[ctx.arg->pairChannelIdxR2], dst, src, ctx.dataSize, ctx.event, 1));
222+ 
223+ ccu::LocalAddr srcSecond;
224+ srcSecond.addr = ctx.output;
225+ srcSecond.addr += ctx.collectPartSize;
226+ srcSecond.addr += ctx.dataSize;
227+ srcSecond.token = ctx.outputToken;
228+ 
229+ ccu::RemoteAddr dstSecond;
230+ dstSecond.addr = ctx.remoteOutput[ctx.arg->pairChannelIdxR2];
231+ dstSecond.addr += ctx.collectPartSize;
232+ dstSecond.addr += ctx.dataSize;
233+ dstSecond.token = ctx.remoteToken[ctx.arg->pairChannelIdxR2];
234+ 
235+ RETURN_IF_CCU_ERROR(
236+ ccu::Write(ctx.arg->channels[ctx.arg->pairChannelIdxR2], dstSecond, srcSecond, ctx.dataSize, ctx.event, 2));
237+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, 3));
238+ return CCU_SUCCESS;
239+ }
240+ 
241+ CcuResult DirectTransfer(DirectPushContext &ctx)
242+ {
243+ return DirectTransferWhole(ctx);
244+ }
245+ 
246+ // Phase-1(layer-0):服内 allgather 前 X = A+D 字节(A 供 phase-2 转发,D 供服内分发)
247+ CcuResult PhaseOneLocal(DirectPushContext &ctx)
248+ {
249+ ccu::Variable gatherSize;
250+ gatherSize = ctx.collectPartSize;
251+ gatherSize += ctx.directPartSize;
252+ 
253+ ccu::LocalAddr src;
254+ src.addr = ctx.input;
255+ src.token = ctx.inputToken;
256+ 
257+ uint32_t operationCount = 0;
258+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
259+ ccu::RemoteAddr dst;
260+ dst.addr = ctx.remoteOutput[channelIdx];
261+ dst.addr += ctx.outputOffset;
262+ dst.token = ctx.remoteToken[channelIdx];
263+ RETURN_IF_CCU_ERROR(ccu::Write(ctx.arg->channels[channelIdx], dst, src, gatherSize, ctx.event,
264+ static_cast<uint16_t>(1U << operationCount)));
265+ ++operationCount;
266+ }
267+ ccu::LocalAddr localDst;
268+ localDst.addr = ctx.output;
269+ localDst.addr += ctx.outputOffset;
270+ localDst.token = ctx.outputToken;
271+ RETURN_IF_CCU_ERROR(ccu::LocalCopy(
272+ localDst, src, gatherSize, ctx.event, static_cast<uint16_t>(1U << operationCount)));
273+ ++operationCount;
274+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, GetCompleteMask(operationCount)));
275+ return CCU_SUCCESS;
276+ }
277+ 
278+ // Phase-1(layer-1):D 段扁平直推全部远端对端 + 尾段 S2 写 pair(由 pair 服内广播)
279+ CcuResult PhaseOneRemote(DirectPushContext &ctx)
280+ {
281+ // D 段直推
282+ ccu::LocalAddr srcDirect;
283+ srcDirect.addr = ctx.input;
284+ srcDirect.addr += ctx.collectPartSize;
285+ srcDirect.token = ctx.inputToken;
286+ 
287+ uint32_t operationCount = 0;
288+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
289+ ccu::RemoteAddr dst;
290+ dst.addr = ctx.remoteOutput[channelIdx];
291+ dst.addr += ctx.outputOffset;
292+ dst.addr += ctx.collectPartSize;
293+ dst.token = ctx.remoteToken[channelIdx];
294+ RETURN_IF_CCU_ERROR(ccu::Write(ctx.arg->channels[channelIdx], dst, srcDirect, ctx.directPartSize,
295+ ctx.event, static_cast<uint16_t>(1U << operationCount)));
296+ ++operationCount;
297+ }
298+ // 尾段 S2 → pair
299+ ccu::Variable relayOffset;
300+ relayOffset = ctx.collectPartSize;
301+ relayOffset += ctx.directPartSize;
302+ 
303+ ccu::LocalAddr srcRelay;
304+ srcRelay.addr = ctx.input;
305+ srcRelay.addr += relayOffset;
306+ srcRelay.token = ctx.inputToken;
307+ 
308+ ccu::RemoteAddr pairDst;
309+ pairDst.addr = ctx.remoteOutput[ctx.arg->pairChannelIdx];
310+ pairDst.addr += ctx.outputOffset;
311+ pairDst.addr += relayOffset;
312+ pairDst.token = ctx.remoteToken[ctx.arg->pairChannelIdx];
313+ RETURN_IF_CCU_ERROR(ccu::Write(ctx.arg->channels[ctx.arg->pairChannelIdx], pairDst, srcRelay,
314+ ctx.relayPartSize, ctx.event, static_cast<uint16_t>(1U << operationCount)));
315+ ++operationCount;
316+ // own 尾段本地落盘(phase-2 服内广播以本端 recvBuf 为源)
317+ ccu::LocalAddr localRelayDst;
318+ localRelayDst.addr = ctx.output;
319+ localRelayDst.addr += ctx.outputOffset;
320+ localRelayDst.addr += relayOffset;
321+ localRelayDst.token = ctx.outputToken;
322+ RETURN_IF_CCU_ERROR(ccu::LocalCopy(localRelayDst, srcRelay, ctx.relayPartSize, ctx.event,
323+ static_cast<uint16_t>(1U << operationCount)));
324+ ++operationCount;
325+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, GetCompleteMask(operationCount)));
326+ return CCU_SUCCESS;
327+ }
328+ 
329+ // Phase-2(layer-1):把服内收集的全部 A 段转发给指定目标(pair/多目标)
330+ CcuResult PhaseTwoRemote(DirectPushContext &ctx)
331+ {
332+ uint32_t operationCount = 0;
333+ for (uint32_t targetIdx = 0; targetIdx < ctx.arg->stageTwoTargetCount; ++targetIdx) {
334+ const uint32_t channelIdx = ctx.arg->stageTwoTargetChannelIdx[targetIdx];
335+ ccu::Variable rankOffset;
336+ rankOffset = ctx.localBlockOffset;
337+ for (uint32_t rankIdx = 0; rankIdx < ctx.arg->localRankCount; ++rankIdx) {
338+ ccu::LocalAddr src;
339+ src.addr = ctx.output;
340+ src.addr += rankOffset;
341+ src.token = ctx.outputToken;
342+ 
343+ ccu::RemoteAddr dst;
344+ dst.addr = ctx.remoteOutput[channelIdx];
345+ dst.addr += rankOffset;
346+ dst.token = ctx.remoteToken[channelIdx];
347+ 
348+ RETURN_IF_CCU_ERROR(ccu::Write(ctx.arg->channels[channelIdx], dst, src, ctx.collectPartSize,
349+ ctx.event, static_cast<uint16_t>(1U << operationCount)));
350+ ++operationCount;
351+ rankOffset += ctx.dataSize;
352+ }
353+ }
354+ if (operationCount != 0) {
355+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, GetCompleteMask(operationCount)));
356+ }
357+ return CCU_SUCCESS;
358+ }
359+ 
360+ // Phase-2(layer-0):服内广播 own 尾段 + 各 owned-remote 的尾段
361+ // rankOffset 为切片起始偏移,尾段在切片内的偏移为 A + D
362+ CcuResult BroadcastRelayPart(DirectPushContext &ctx, ccu::Variable rankOffset, uint32_t &operationCount)
363+ {
364+ ccu::Variable relayOffset;
365+ relayOffset = ctx.collectPartSize;
366+ relayOffset += ctx.directPartSize;
367+ 
368+ ccu::LocalAddr src;
369+ src.addr = ctx.output;
370+ src.addr += rankOffset;
371+ src.addr += relayOffset;
372+ src.token = ctx.outputToken;
373+ 
374+ for (uint32_t channelIdx = 0; channelIdx < ctx.arg->channelCount; ++channelIdx) {
375+ ccu::RemoteAddr dst;
376+ dst.addr = ctx.remoteOutput[channelIdx];
377+ dst.addr += rankOffset;
378+ dst.addr += relayOffset;
379+ dst.token = ctx.remoteToken[channelIdx];
380+ RETURN_IF_CCU_ERROR(ccu::Write(ctx.arg->channels[channelIdx], dst, src, ctx.relayPartSize, ctx.event,
381+ static_cast<uint16_t>(1U << operationCount)));
382+ ++operationCount;
383+ }
384+ return CCU_SUCCESS;
385+ }
386+ 
387+ CcuResult PhaseTwoLocal(DirectPushContext &ctx)
388+ {
389+ uint32_t operationCount = 0;
390+ // own 尾段(从本端 recvBuf 取,phase-1 已由 LocalCopy 落盘)
391+ RETURN_IF_CCU_ERROR(BroadcastRelayPart(ctx, ctx.outputOffset, operationCount));
392+ // 各 owned-remote 尾段(phase-1 经 Clos 到达本端 recvBuf)
393+ for (uint32_t ownedIdx = 0; ownedIdx < ctx.arg->ownedRemoteCount; ++ownedIdx) {
394+ ccu::Variable ownedOffset;
395+ ownedOffset = 0;
396+ for (uint32_t offsetIdx = 0; offsetIdx < ctx.arg->ownedRemoteRanks[ownedIdx]; ++offsetIdx) {
397+ ownedOffset += ctx.dataSize;
398+ }
399+ RETURN_IF_CCU_ERROR(BroadcastRelayPart(ctx, ownedOffset, operationCount));
400+ }
401+ RETURN_IF_CCU_ERROR(ccu::EventWait(ctx.event, GetCompleteMask(operationCount)));
402+ return CCU_SUCCESS;
403+ }
404+ 
405+ CcuResult Transfer(DirectPushContext &ctx)
406+ {
407+ CCU_IF(ctx.mode == TRANSFER_PHASE_ONE)
408+ {
409+ if (ctx.arg->isLocalLayer != 0) {
410+ RETURN_IF_CCU_ERROR(PhaseOneLocal(ctx));
411+ } else {
412+ RETURN_IF_CCU_ERROR(PhaseOneRemote(ctx));
413+ }
414+ }
415+ CCU_ELSE
416+ {
417+ CCU_IF(ctx.mode == TRANSFER_PHASE_TWO)
418+ {
419+ if (ctx.arg->isLocalLayer != 0) {
420+ RETURN_IF_CCU_ERROR(PhaseTwoLocal(ctx));
421+ } else {
422+ RETURN_IF_CCU_ERROR(PhaseTwoRemote(ctx));
423+ }
424+ }
425+ CCU_ELSE
426+ {
427+ RETURN_IF_CCU_ERROR(DirectTransfer(ctx));
428+ }
429+ }
430+ return CCU_SUCCESS;
431+ }
432+ 
433+} // namespace
434+ 
435+CcuResult CcuAllGatherDirectPush(CcuKernelArg arg)
436+{
437+ auto *kernelArg = static_cast<CcuKernelArgAllGather *>(arg);
438+ if (kernelArg == nullptr || kernelArg->channelCount == 0 || kernelArg->channelCount >= MAX_RANK_SIZE) {
439+ HCCL_ERROR("[AllGather] Invalid CCU kernel argument");
440+ return CCU_E_PARA;
441+ }
442+ 
443+ DirectPushContext ctx;
444+ ctx.arg = kernelArg;
445+ RETURN_IF_CCU_ERROR(InitResources(ctx));
446+ RETURN_IF_CCU_ERROR(LoadTaskArgs(ctx));
447+ CCU_IF(ctx.firstCall != 0)
448+ {
449+ RETURN_IF_CCU_ERROR(ExchangeRemoteOutput(ctx));
450+ }
451+ CCU_IF(ctx.mode == TRANSFER_FLAT)
452+ {
453+ // 小消息扁平整写:信用后移——先等上一轮(奇偶相反位)的 post,再传数据,
454+ // 本轮只 record 不 wait;首轮(firstCall)无上一轮可等
455+ CCU_IF(ctx.firstCall == 0)
456+ {
457+ CCU_IF(ctx.syncParity == 0)
458+ {
459+ RETURN_IF_CCU_ERROR(PostWaitOnly(ctx, POST_MASK_ODD));
460+ }
461+ CCU_ELSE
462+ {
463+ RETURN_IF_CCU_ERROR(PostWaitOnly(ctx, POST_MASK_EVEN));
464+ }
465+ }
466+ RETURN_IF_CCU_ERROR(DirectTransfer(ctx));
467+ CCU_IF(ctx.syncParity == 0)
468+ {
469+ RETURN_IF_CCU_ERROR(PostRecordOnly(ctx, POST_MASK_EVEN));
470+ }
471+ CCU_ELSE
472+ {
473+ RETURN_IF_CCU_ERROR(PostRecordOnly(ctx, POST_MASK_ODD));
474+ }
475+ }
476+ CCU_ELSE
477+ {
478+ RETURN_IF_CCU_ERROR(Transfer(ctx));
479+ CCU_IF(ctx.syncParity == 0)
480+ {
481+ RETURN_IF_CCU_ERROR(PostSync(ctx, POST_MASK_EVEN));
482+ }
483+ CCU_ELSE
484+ {
485+ RETURN_IF_CCU_ERROR(PostSync(ctx, POST_MASK_ODD));
486+ }
487+ }
488+ return CCU_SUCCESS;
489+}
490+ 
491+namespace {
492+ 
493+ // 极简扁平 kernel 的 taskArgs(10 个):与 CcuAllGatherFlat 的 LoadArg 一一对应
494+ CcuResult FlatLoadTaskArgs(DirectPushContext &ctx)
495+ {
496+ uint32_t argIdx = 0;
497+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.input, argIdx++));
498+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.output, argIdx++));
499+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.inputToken, argIdx++));
500+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.outputToken, argIdx++));
501+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.dataSize, argIdx++));
502+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.outputOffset, argIdx++));
503+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.mode, argIdx++));
504+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.collectPartSize, argIdx++)); // round-2:pairBaseOffset
505+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.firstCall, argIdx++)); // 1 = 本轮先显式交换地址/token
506+ RETURN_IF_CCU_ERROR(ccu::LoadArg(ctx.syncParity, argIdx++));
507+ return CCU_SUCCESS;
508+ }
509+ 
510+} // namespace
511+ 
512+// 极简扁平 kernel:小消息与 4*1 拓扑;分层 kernel 之外的独立注册,指令数最小化
513+CcuResult CcuAllGatherFlat(CcuKernelArg arg)
514+{
515+ auto *kernelArg = static_cast<CcuKernelArgAllGather *>(arg);
516+ if (kernelArg == nullptr || kernelArg->channelCount == 0 || kernelArg->channelCount >= MAX_RANK_SIZE) {
517+ HCCL_ERROR("[AllGather] Invalid CCU kernel argument");
518+ return CCU_E_PARA;
519+ }
520+ 
521+ DirectPushContext ctx;
522+ ctx.arg = kernelArg;
523+ RETURN_IF_CCU_ERROR(InitResources(ctx));
524+ RETURN_IF_CCU_ERROR(FlatLoadTaskArgs(ctx));
525+ CCU_IF(ctx.firstCall != 0)
526+ {
527+ RETURN_IF_CCU_ERROR(ExchangeRemoteOutput(ctx));
528+ }
529+ CCU_IF(ctx.mode == TRANSFER_ROUND_FUSED)
530+ {
531+ // 4*1 递归倍增单 launch 版:round1(r^1,own S)→ 内部 barrier →
532+ // round2(r^2,相邻两片 2S)→ 收尾 barrier;两个 barrier 分用 even/odd 位
533+ RETURN_IF_CCU_ERROR(RoundOneTransfer(ctx));
534+ RETURN_IF_CCU_ERROR(PostSync(ctx, POST_MASK_EVEN));
535+ RETURN_IF_CCU_ERROR(RoundTwoTransfer(ctx));
536+ RETURN_IF_CCU_ERROR(PostSync(ctx, POST_MASK_ODD));
537+ }
538+ CCU_ELSE
539+ {
540+ // 小消息扁平整写:信用后移——先等上一轮(奇偶相反位)的 post,再传数据,
541+ // 本轮只 record 不 wait;首轮(firstCall)无上一轮可等
542+ CCU_IF(ctx.firstCall == 0)
543+ {
544+ CCU_IF(ctx.syncParity == 0)
545+ {
546+ RETURN_IF_CCU_ERROR(PostWaitOnly(ctx, POST_MASK_ODD));
547+ }
548+ CCU_ELSE
549+ {
550+ RETURN_IF_CCU_ERROR(PostWaitOnly(ctx, POST_MASK_EVEN));
551+ }
552+ }
553+ RETURN_IF_CCU_ERROR(DirectTransferWhole(ctx));
554+ CCU_IF(ctx.syncParity == 0)
555+ {
556+ RETURN_IF_CCU_ERROR(PostRecordOnly(ctx, POST_MASK_EVEN));
557+ }
558+ CCU_ELSE
559+ {
560+ RETURN_IF_CCU_ERROR(PostRecordOnly(ctx, POST_MASK_ODD));
561+ }
562+ }
563+ return CCU_SUCCESS;
564+}
565+ 
566+} // namespace ops_hccl
567+ 
568+#undef RETURN_IF_CCU_ERROR
@@ -0,0 +1,25 @@
1+/**
2+ * Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+ * This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+ * CANN Open Software License Agreement Version 2.0 (the "License").
5+ * Please refer to the License for details. You may not use this file except in compliance with the License.
6+ * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+ * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+ * See LICENSE in the root of the software repository for the full text of the License.
9+ */
10+ 
11+#ifndef OPS_HCCL_CCU_KERNEL_H
12+#define OPS_HCCL_CCU_KERNEL_H
13+ 
14+#include <ccu/ccu_types.h>
15+ 
16+namespace ops_hccl {
17+ 
18+// 小包及 4x1 拓扑使用直达 push;2x8 / 8+4 大包把数据沿两个维度拆分,
19+// 由同一函数注册出的 layer-0 / layer-1 kernel 并行执行两个阶段。
20+CcuResult CcuAllGatherDirectPush(CcuKernelArg arg);
21+// 极简扁平 kernel:小消息与 4*1 拓扑(独立注册,指令数最小化)
22+CcuResult CcuAllGatherFlat(CcuKernelArg arg);
23+} // namespace ops_hccl
24+ 
25+#endif // OPS_HCCL_CCU_KERNEL_H
@@ -0,0 +1,106 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+Language: Cpp
12+Standard: c++17
13+TabWidth: 4
14+UseTab: Never
15+UseCRLF: false
16+AccessModifierOffset: -4
17+AlignConsecutiveAssignments: false
18+AlignConsecutiveDeclarations: false
19+AlignEscapedNewlines: DontAlign
20+AlignOperands: true
21+AlignTrailingComments: true
22+AllowAllArgumentsOnNextLine: true
23+AllowShortLambdasOnASingleLine: Empty
24+AllowAllParametersOfDeclarationOnNextLine: false
25+AllowShortBlocksOnASingleLine: false
26+AllowShortCaseLabelsOnASingleLine: false
27+AllowShortFunctionsOnASingleLine: false
28+AllowShortIfStatementsOnASingleLine: false
29+AllowShortLoopsOnASingleLine: false
30+AlwaysBreakAfterDefinitionReturnType: None
31+AlwaysBreakBeforeMultilineStrings: false
32+AlwaysBreakTemplateDeclarations: MultiLine
33+BinPackArguments: true
34+BinPackParameters: true
35+AlignAfterOpenBracket: DontAlign
36+ 
37+BraceWrapping:
38+ AfterCaseLabel: false
39+ AfterClass: false
40+ AfterControlStatement: Never
41+ AfterEnum: false
42+ AfterFunction: true
43+ AfterNamespace: false
44+ AfterStruct: false
45+ AfterUnion: false
46+ AfterExternBlock: false
47+ BeforeCatch: false
48+ BeforeElse: false
49+ IndentBraces: false
50+ SplitEmptyFunction: true
51+ SplitEmptyRecord: true
52+ SplitEmptyNamespace: true
53+ 
54+BreakBeforeBinaryOperators: All
55+BreakBeforeBraces: Custom
56+BreakBeforeTernaryOperators: true
57+BreakConstructorInitializersBeforeComma: false
58+BreakInheritanceList: AfterColon
59+ColumnLimit: 120
60+CommentPragmas: '^ IWYU pragma:'
61+PackConstructorInitializers: CurrentLine
62+ConstructorInitializerIndentWidth: 4
63+ContinuationIndentWidth: 4
64+Cpp11BracedListStyle: true
65+DerivePointerAlignment: false
66+DisableFormat: false
67+ExperimentalAutoDetectBinPacking: false
68+ForEachMacros: [ foreach ]
69+ 
70+IncludeCategories:
71+ - Regex: '^<'
72+ Priority: 3
73+ - Regex: '^"hccl'
74+ Priority: 2
75+ - Regex: '.*'
76+ Priority: 1
77+ 
78+IndentCaseLabels: true
79+IndentWidth: 4
80+IndentWrappedFunctionNames: false
81+KeepEmptyLinesAtTheStartOfBlocks: false
82+MacroBlockBegin: ''
83+MacroBlockEnd: ''
84+MaxEmptyLinesToKeep: 1
85+NamespaceIndentation: Inner
86+ 
87+PenaltyBreakBeforeFirstCallParameter: 19
88+PenaltyBreakComment: 300
89+PenaltyBreakFirstLessLess: 120
90+PenaltyBreakString: 1000
91+PenaltyExcessCharacter: 1000000
92+PenaltyReturnTypeOnItsOwnLine: 60
93+ 
94+PointerAlignment: Right
95+ReflowComments: true
96+SortIncludes: false
97+SpaceAfterCStyleCast: false
98+SpaceBeforeAssignmentOperators: true
99+SpaceBeforeParens: ControlStatements
100+SpaceInEmptyParentheses: false
101+SpacesBeforeTrailingComments: 1
102+SpacesInAngles: false
103+SpacesInContainerLiterals: true
104+SpacesInCStyleCastParentheses: false
105+SpacesInParentheses: false
106+SpacesInSquareBrackets: false
@@ -0,0 +1,11 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+* text=auto eol=lf
@@ -0,0 +1,35 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+# IDE
12+.vscode/
13+.idea/
14+.cache/
15+.DS_Store
16+*~
17+*.swp
18+*.swo
19+ 
20+# Build
21+build/
22+build_ut/
23+third_party/
24+__pycache__/
25+ 
26+# Agent
27+.omo/
28+.claude/
29+.opencode/
30+CLAUDE.md
31+AGENTS.md
32+ 
33+# Logs
34+logs/
35+*.log
@@ -0,0 +1,47 @@
1+# -----------------------------------------------------------------------------------------------------------
2+# Copyright (c) 2026 Huawei Technologies Co., Ltd.
3+# This program is free software, you can redistribute it and/or modify it under the terms and conditions of
4+# CANN Open Software License Agreement Version 2.0 (the "License").
5+# Please refer to the License for details. You may not use this file except in compliance with the License.
6+# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED,
7+# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
8+# See LICENSE in the root of the software repository for the full text of the License.
9+# -----------------------------------------------------------------------------------------------------------
10+ 
11+cmake_minimum_required(VERSION 3.16.0)
12+project(hccl_allgather_aicpu)
13+ 
14+message(STATUS "CMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}")
15+message(STATUS "ASCEND_CANN_PACKAGE_PATH=${ASCEND_CANN_PACKAGE_PATH}")
16+ 
17+set(HCC_TOOLCHAIN_DIR "${ASCEND_CANN_PACKAGE_PATH}/toolkit/toolchain/hcc")
18+set(HCC_CXX_COMPILER "${HCC_TOOLCHAIN_DIR}/bin/aarch64-target-linux-gnu-g++")
19+set(HCC_C_COMPILER "${HCC_TOOLCHAIN_DIR}/bin/aarch64-target-linux-gnu-gcc")
20+set(HCC_C_AR "${HCC_TOOLCHAIN_DIR}/bin/aarch64-target-linux-gnu-ar")
21+ 
22+# 编译 Host 侧链接库
23+add_subdirectory(op_host)
24+ 
25+# 编译 Device 侧链接库
26+include(ExternalProject)
27+ExternalProject_Add(hccl_device
28+ SOURCE_DIR ${CMAKE_SOURCE_DIR}/op_kernel_aicpu
29+ BINARY_DIR ${CMAKE_BINARY_DIR}/device_build
30+ CMAKE_ARGS
31+ -DTOOLCHAIN_DIR=${HCC_TOOLCHAIN_DIR}
32+ -DCMAKE_C_COMPILER=${HCC_C_COMPILER}
33+ -DCMAKE_CXX_COMPILER=${HCC_CXX_COMPILER}
34+ -DCMAKE_C_COMPILER_LAUNCHER=${CMAKE_C_COMPILER_LAUNCHER}
35+ -DCMAKE_CXX_COMPILER_LAUNCHER=${CMAKE_CXX_COMPILER_LAUNCHER}
36+ -DCMAKE_C_AR=${HCC_C_AR}
37+ -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
38+ -DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX}
39+ -DASCEND_CANN_PACKAGE_PATH=${ASCEND_CANN_PACKAGE_PATH}
40+ INSTALL_COMMAND ${CMAKE_COMMAND} --install ${CMAKE_BINARY_DIR}/device_build --prefix ${CMAKE_INSTALL_PREFIX}
41+ BUILD_ALWAYS TRUE
42+)
43+ 
44+# 安装
45+install(FILES include/hccl.h
46+ DESTINATION include
47+)
@@ -0,0 +1,201 @@
1+ Apache License
2+ Version 2.0, January 2004
3+ http://www.apache.org/licenses/
4+ 
5+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6+ 
7+ 1. Definitions.
8+ 
9+ "License" shall mean the terms and conditions for use, reproduction,
10+ and distribution as defined by Sections 1 through 9 of this document.
11+ 
12+ "Licensor" shall mean the copyright owner or entity authorized by
13+ the copyright owner that is granting the License.
14+ 
15+ "Legal Entity" shall mean the union of the acting entity and all
16+ other entities that control, are controlled by, or are under common
17+ control with that entity. For the purposes of this definition,
18+ "control" means (i) the power, direct or indirect, to cause the
19+ direction or management of such entity, whether by contract or
20+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21+ outstanding shares, or (iii) beneficial ownership of such entity.
22+ 
23+ "You" (or "Your") shall mean an individual or Legal Entity
24+ exercising permissions granted by this License.
25+ 
26+ "Source" form shall mean the preferred form for making modifications,
27+ including but not limited to software source code, documentation
28+ source, and configuration files.
29+ 
30+ "Object" form shall mean any form resulting from mechanical
31+ transformation or translation of a Source form, including but
32+ not limited to compiled object code, generated documentation,
33+ and conversions to other media types.
34+ 
35+ "Work" shall mean the work of authorship, whether in Source or
36+ Object form, made available under the License, as indicated by a
37+ copyright notice that is included in or attached to the work
38+ (an example is provided in the Appendix below).
39+ 
40+ "Derivative Works" shall mean any work, whether in Source or Object
41+ form, that is based on (or derived from) the Work and for which the
42+ editorial revisions, annotations, elaborations, or other modifications
43+ represent, as a whole, an original work of authorship. For the purposes
44+ of this License, Derivative Works shall not include works that remain
45+ separable from, or merely link (or bind by name) to the interfaces of,
46+ the Work and Derivative Works thereof.
47+ 
48+ "Contribution" shall mean any work of authorship, including
49+ the original version of the Work and any modifications or additions
50+ to that Work or Derivative Works thereof, that is intentionally
51+ submitted to Licensor for inclusion in the Work by the copyright owner
52+ or by an individual or Legal Entity authorized to submit on behalf of
53+ the copyright owner. For the purposes of this definition, "submitted"
54+ means any form of electronic, verbal, or written communication sent
55+ to the Licensor or its representatives, including but not limited to
56+ communication on electronic mailing lists, source code control systems,
57+ and issue tracking systems that are managed by, or on behalf of, the
58+ Licensor for the purpose of discussing and improving the Work, but
59+ excluding communication that is conspicuously marked or otherwise
60+ designated in writing by the copyright owner as "Not a Contribution."
61+ 
62+ "Contributor" shall mean Licensor and any individual or Legal Entity
63+ on behalf of whom a Contribution has been received by Licensor and
64+ subsequently incorporated within the Work.
65+ 
66+ 2. Grant of Copyright License. Subject to the terms and conditions of
67+ this License, each Contributor hereby grants to You a perpetual,
68+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
69+ copyright license to reproduce, prepare Derivative Works of,
70+ publicly display, publicly perform, sublicense, and distribute the
71+ Work and such Derivative Works in Source or Object form.
72+ 
73+ 3. Grant of Patent License. Subject to the terms and conditions of
74+ this License, each Contributor hereby grants to You a perpetual,
75+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
76+ (except as stated in this section) patent license to make, have made,
77+ use, offer to sell, sell, import, and otherwise transfer the Work,
78+ where such license applies only to those patent claims licensable
79+ by such Contributor that are necessarily infringed by their
80+ Contribution(s) alone or by combination of their Contribution(s)
81+ with the Work to which such Contribution(s) was submitted. If You
82+ institute patent litigation against any entity (including a
83+ cross-claim or counterclaim in a lawsuit) alleging that the Work
84+ or a Contribution incorporated within the Work constitutes direct
85+ or contributory patent infringement, then any patent licenses
86+ granted to You under this License for that Work shall terminate
87+ as of the date such litigation is filed.
88+ 
89+ 4. Redistribution. You may reproduce and distribute copies of the
90+ Work or Derivative Works thereof in any medium, with or without
91+ modifications, and in Source or Object form, provided that You
92+ meet the following conditions:
93+ 
94+ (a) You must give any other recipients of the Work or
95+ Derivative Works a copy of this License; and
96+ 
97+ (b) You must cause any modified files to carry prominent notices
98+ stating that You changed the files; and
99+ 
100+ (c) You must retain, in the Source form of any Derivative Works
101+ that You distribute, all copyright, patent, trademark, and
102+ attribution notices from the Source form of the Work,
103+ excluding those notices that do not pertain to any part of
104+ the Derivative Works; and
105+ 
106+ (d) If the Work includes a "NOTICE" text file as part of its
107+ distribution, then any Derivative Works that You distribute must
108+ include a readable copy of the attribution notices contained
109+ within such NOTICE file, excluding those notices that do not
110+ pertain to any part of the Derivative Works, in at least one
111+ of the following places: within a NOTICE text file distributed
112+ as part of the Derivative Works; within the Source form or
113+ documentation, if provided along with the Derivative Works; or,
114+ within a display generated by the Derivative Works, if and
115+ wherever such third-party notices normally appear. The contents
116+ of the NOTICE file are for informational purposes only and
117+ do not modify the License. You may add Your own attribution
118+ notices within Derivative Works that You distribute, alongside
119+ or as an addendum to the NOTICE text from the Work, provided
120+ that such additional attribution notices cannot be construed
121+ as modifying the License.
122+ 
123+ You may add Your own copyright statement to Your modifications and
124+ may provide additional or different license terms and conditions
125+ for use, reproduction, or distribution of Your modifications, or
126+ for any such Derivative Works as a whole, provided Your use,
127+ reproduction, and distribution of the Work otherwise complies with
128+ the conditions stated in this License.
129+ 
130+ 5. Submission of Contributions. Unless You explicitly state otherwise,
131+ any Contribution intentionally submitted for inclusion in the Work
132+ by You to the Licensor shall be under the terms and conditions of
133+ this License, without any additional terms or conditions.
134+ Notwithstanding the above, nothing herein shall supersede or modify
135+ the terms of any separate license agreement you may have executed
136+ with Licensor regarding such Contributions.
137+ 
138+ 6. Trademarks. This License does not grant permission to use the trade
139+ names, trademarks, service marks, or product names of the Licensor,
140+ except as required for reasonable and customary use in describing the
141+ origin of the Work and reproducing the content of the NOTICE file.
142+ 
143+ 7. Disclaimer of Warranty. Unless required by applicable law or
144+ agreed to in writing, Licensor provides the Work (and each
145+ Contributor provides its Contributions) on an "AS IS" BASIS,
146+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
147+ implied, including, without limitation, any warranties or conditions
148+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
149+ PARTICULAR PURPOSE. You are solely responsible for determining the
150+ appropriateness of using or redistributing the Work and assume any
151+ risks associated with Your exercise of permissions under this License.
152+ 
153+ 8. Limitation of Liability. In no event and under no legal theory,
154+ whether in tort (including negligence), contract, or otherwise,
155+ unless required by applicable law (such as deliberate and grossly
156+ negligent acts) or agreed to in writing, shall any Contributor be
157+ liable to You for damages, including any direct, indirect, special,
158+ incidental, or consequential damages of any character arising as a
159+ result of this License or out of the use or inability to use the
160+ Work (including but not limited to damages for loss of goodwill,
161+ work stoppage, computer failure or malfunction, or any and all
162+ other commercial damages or losses), even if such Contributor
163+ has been advised of the possibility of such damages.
164+ 
165+ 9. Accepting Warranty or Additional Liability. While redistributing
166+ the Work or Derivative Works thereof, You may choose to offer,
167+ and charge a fee for, acceptance of support, warranty, indemnity,
168+ or other liability obligations and/or rights consistent with this
169+ License. However, in accepting such obligations, You may act only
170+ on Your own behalf and on Your sole responsibility, not on behalf
171+ of any other Contributor, and only if You agree to indemnify,
172+ defend, and hold each Contributor harmless for any liability
173+ incurred by, or claims asserted against, such Contributor by reason
174+ of your accepting any such warranty or additional liability.
175+ 
176+ END OF TERMS AND CONDITIONS
177+ 
178+ APPENDIX: How to apply the Apache License to your work.
179+ 
180+ To apply the Apache License to your work, attach the following
181+ boilerplate notice, with the fields enclosed by brackets "{}"
182+ replaced with your own identifying information. (Don't include
183+ the brackets!) The text should be enclosed in the appropriate
184+ comment syntax for the file format. We also recommend that a
185+ file or class name and description of purpose be included on the
186+ same "printed page" as the copyright notice for easier
187+ identification within third-party archives.
188+ 
189+ Copyright {yyyy} {name of copyright owner}
190+ 
191+ Licensed under the Apache License, Version 2.0 (the "License");
192+ you may not use this file except in compliance with the License.
193+ You may obtain a copy of the License at
194+ 
195+ http://www.apache.org/licenses/LICENSE-2.0
196+ 
197+ Unless required by applicable law or agreed to in writing, software
198+ distributed under the License is distributed on an "AS IS" BASIS,
199+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
200+ See the License for the specific language governing permissions and
201+ limitations under the License.